Messages

Another solution is to get a Micron MX300 (and specifically that model) drive - even a 512GB one will do. Then use Micron's Executive software to over-provision the drive by 50% i.e. it's a 256GB drive for all intents and purposes. The drive will now be able to take 3-4 DWPD (drive writes per day) even without TRIM. You should check if TRIM is enabled for your filesystem - are you using ZFS or UFS?

Thank you - that one is indeed more reasonably priced.

"drive will now be able to take 3-4 DWPD (drive writes per day) even without TRIM" - does that mean that it can take 3-4 times its provisioned capacity (3-4x256GB)? Or am I misunderstanding?

"You should check if TRIM is enabled for your filesystem - are you using ZFS or UFS?" - I use ZFS (for snapshotting), and have enabled trim in cron:

Is that what you meant? Thanks :)

Thank you ALL for your answers, especially on a Sunday. This is a wonderful community, which is commendable!

1. the almost-dead drive is a Samsung 256GB. I unfortunately didn't do a S.M.A.R.T on it prior to deploying it in my "first" opnsense attempt, but I think it came from a desktop or laptop (a friend gave it to me years ago, he used to work in a company that scrapped all hardware after 3 years, and I had the disk in storage for a few years). So conceivably it might have been "tired" when I deployed it, but I highly doubt that it was written onto at high volumes - those companies lock down workstations and all files are hosted on servers (or the cloud, nowadays). However it's likely that I broke it by letting it getting constantly hammered by opnsense and it's add-ons :)

2. I have an 128GB Intel 520 Series, which appears to have enjoyed Industry-leading 5-year warranty and historically low failure/return rates when released (in 2012, so it's also not a spring chicken). But I did run a S.M.A.R.T on it, and it reports 96% health and ~10,000 hours of uptime (which is a lot, but perhaps no big deal). I did install opnsense on it temporarily, but decided to go with a bigger drive (512GB, brandnew, but a crappy brand - I am monitoring it, and have all backups to date). But I am considering deploying that one, if I see that the "crappy 512GB" starts going downhill fast. I will also keep my eyes open for an enterprise-grade SSD...it's getting harder now, with everyone going to the cloud...in the past I literally had access to mountains of untouched never-to-be-deployed enterprise-grade equipment headed for the bin.

3. at work we log about 25TB/day (not a typo), which we're now in the process of consolidating to elastic - so I'm quite aware of the challenges when ingesting such high log volumes - and which was also one of the reasons I might have gone a bit overboard when enabling too much logging on my opnsense - it was regularly logging over 1mil entries/day. I have now redone the ZA configuration, and am now protecting/logging individual VLANs(about 8, for segmentation)/NICs(I have 5 of them, a quad and the onboard one) instead of latching ZA onto the trunk port. This allowed me to avoid assessing&logging entries from the most verbose host on my network - which due to the very high volume would have been very hard to qualitatively analyze anyway. I can however re-enable it with one click if necessary.

4. getting back to the elastic architecture, IMO it would have been great if I could have assigned some GBs as a ram disk, and for elastic to log to its heart's content, entries to be pruned when they exceed the assigned RAM quantity. I could have lived (privately, unlike at work where it's highly regulated) with potentially missing out on older entries - but it would have been great for the longevity of my OS drive. Or another decent option (perhaps that is possible) would be if I could add another drive, and to redirect all elastic logs to it - it's much less bad if I lose my logs, but the FW continues with all its tasks (again, at home that would be acceptable, at work we have everything set to stop forwarding if it can be logged). I know I can log to an external host, but I would like to keep the "firewall service" to one host if possible. I used to rely on one of the major enterprise firewalls (licensed, promotionally) prior to deploying opnsense, so I am mindful of the form factor. But I really love opnsense so I want to carry on <3

5. is disabling hostwatch still a thing? I noticed it was a considered an issue in the past, but Frank suggested it's been fixed in Jan'26 - I am using the most recent version. I have about 30-40 devices altogether, and aside from the mobile devices which tend to use random MACs, all else is supposed to be very stable over time. How can I check how much hostwatch has been logging so far?

6. in conclusion, I consider this experience a great learning opportunity. I encountered a couple of challenges when restoring opnsense (external to opnsense itself), which I have found solutions for - so right now it would take me under 1h to restore, so no big deal. But I am concerned about the FW dying suddenly because I'm sometimes away from home for longer (e.g., during holidays), and I am using it for many reasons, including as a VPN gateway to keep all my travel traffic curated - and I offer friends the same services, so there are several people relying on this 'safe perimeter security', and I don't want them to be disappointed. Also, it is my way to monitor a lot of things at home. It feels like a very no-enterprise way to introduce such a frail SPOF into the equation...my2c

Patrick: na endlich hat es geklappt - danke :)

The disk is filled by zenarmor/elastic. I'll have to further tweak logging. I definitely want the ZA "protection", and would like to be able to take a look at what's happening, but ideally it would store the fresh logs in RAM before flushing them (and I've bumped up space in Reporting Data/Memory Disk Size to 500MB, the maximum permitted).

Update: I have also temporarily stopped ntopng - that one also appears to fill the disk with the timeseries...

@oc Du hast das

Code Select Expand

cd /

vorher vergessen. Nach deinem Prompt zu urteilen bist du immer noch im Home-Directory /root.

Danke Patrick!

root@OC-OPNsense01:/ # du -sk * | sort -rn | head
2866972 usr
1231306 var
174862  boot
11405   tmp
10852   lib
5799    conf
3372    sbin
2489    etc
929     bin
146     libexec

And...according to the dashboard widget, another 100MB has been added to disk since the last message I've posted.

Vielen Dank, Patrick...aber:

root@OC-OPNsense01:~ # du -sk * | sort -rn | head
du: No match.

[what continues to fill up my drive?]

Hello,

After about six months of having deployed my first "production" OPNsense, during a scheduled reboot (replacing the UPS with a larger one) the computer's BIOS indicated that S.M.A.R.T is deeming the drive as "about to fail". I have good backups, and was able to still access the old drive even after I had restored the setup onto the new one (I was initially missing a backup, that of ntopng, which together with zenarmor has its own configuration settings file) - so I can write if off as a good exercise of restoring OPNsense (albeit onto the same computer).

However after having gone through that experience (I never had any of my SSDs die on me, and I own some that I have used for well over ten years - including the one in the laptop I'm writing this message from), and after having more experience with OPNsense than in the beginning, and knowing that I won't spend much time with historical records (for instance I had set up Zenarmor to keep logs for 90 days, I now reduced that to three days), to extend the life of my next SSD I have decided to lighten logging considerably (I had already enabled /var/log RAM disk and /tmp RAM disk). I have disabled netflow, reduced zenarmor logging, ntopng logging, re-enabled the cron for trimming zroot - and started paying close attention to free space and S.M.A.R.T.

So here's my question: what continues to fill up my drive? The dashboard widget reports 0.7GB more used space in less than 24h. Some pointers on what to look for would be really helpful. My (new) mindset is to keep logging disabled as much as possible, and to only enable something if I need it for troubleshooting and/or forensics. But at the moment I'd like to know what's going on, what is writing to the drive - I would like this one to last MUCH longer. Note: I am not a FreeBSD file system expert, but am able to SSH/SCP in the filesystem as necessary.

Thank you :)

First and foremost, my apologies for the late reply. I was finally able to reboot the OPNsense today. It is now updated to 26.7.6

OK, while the condition exists (for recollection, aside from a long boot time because the wireguard service hangs for minutes, wireguard clients connected through the OPNsense don't have DNS resolution until the DNS service is restarted, afterward all works well until the next reboot, and this was linked to having outbound wireguard connections to external hosts with dynamic IPs resolved through URLs - and I've tried with A records with long TTLs rather than the typical 5min offered by dynamic dns - it didn't bring anything - the only (unacceptable) workaround I found to reduce boot time and not trigger the issue was replacing the external wireguard server host names DNS names with their IPs - but I had to revert back to their names, as the IPs can change over time):

root@OC-OPNsense01:~ # ifconfig wg4
wg4: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1420
        description: 127OPNSense_WG_Server (opt15)
        options=180000<LINKSTATE,NETMAP>
        inet 10.127.127.1 netmask 0xffffff00
        groups: wg wireguard
        nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
root@OC-OPNsense01:~ #

root@OC-OPNsense01:~ # wg show wg4
interface: wg4
  public key: *************************************=
  private key: (hidden)
  listening port: 51***

peer: *************************************==
  preshared key: (hidden)
  endpoint: *************************************=:49222
  allowed ips: 10.127.127.2/32
  latest handshake: Now
  transfer: 929.85 KiB received, 10.81 KiB sent
  persistent keepalive: every 25 seconds

peer: *************************************==
  preshared key: (hidden)
  allowed ips: 10.127.127.4/32
  transfer: 0 B received, 11.42 KiB sent
  persistent keepalive: every 25 seconds

peer: *************************************==
  preshared key: (hidden)
  allowed ips: 10.127.127.3/32
  transfer: 0 B received, 11.56 KiB sent
  persistent keepalive: every 25 seconds

peer: *************************************==
  preshared key: (hidden)
  allowed ips: 10.127.127.6/32
  transfer: 0 B received, 11.56 KiB sent
  persistent keepalive: every 25 seconds

peer: *************************************==
  preshared key: (hidden)
  allowed ips: 10.127.127.7/32
  transfer: 0 B received, 11.56 KiB sent
  persistent keepalive: every 25 seconds

peer: *************************************==
  preshared key: (hidden)
  allowed ips: 10.127.127.8/32
  transfer: 0 B received, 11.56 KiB sent
  persistent keepalive: every 25 seconds

peer: *************************************==
  preshared key: (hidden)
  allowed ips: 10.127.127.5/32
  transfer: 0 B received, 11.56 KiB sent
  persistent keepalive: every 25 seconds
root@OC-OPNsense01:~ #

root@OC-OPNsense01:~ # netstat -rn -f inet
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            **********.1       UGS            igb3
1.1.1.1            **********.1       UGHS           igb3
10.6.0.1           link#11            UHS             wg0
10.6.0.5           link#6             UH              lo0
10.10.112.0/24     link#5             U               re0
10.10.112.112      link#6             UHS             lo0
10.14.0.0/16       link#12            U               wg2
10.14.0.1          link#12            UHS             wg2
10.14.0.2          link#6             UHS             lo0
10.127.2.0/24      link#14            U            vlan02
10.127.2.1         link#6             UHS             lo0
10.127.3.0/24      link#15            U            vlan03
10.127.3.1         link#6             UHS             lo0
10.127.4.0/24      link#16            U            vlan04
10.127.4.1         link#6             UHS             lo0
10.127.7.0/24      link#18            U            vlan07
10.127.7.1         link#6             UHS             lo0
10.127.11.0/24     link#19            U           vlan011
10.127.11.1        link#6             UHS             lo0
10.127.12.0/24     link#20            U           vlan012
10.127.12.1        link#6             UHS             lo0
10.127.60.0/24     link#2             U              igb1
10.127.60.1        link#6             UHS             lo0
10.127.127.0/24    link#10            U               wg4
10.127.127.1       link#6             UHS             lo0
**********.      192.168.60.1       UGHS           igb2
127.0.0.1          link#6             UH              lo0
**********.     **********.       UGHS           igb3
**********.     **********.       UGHS           igb3
**********.    10.14.0.1          UGHS            wg2
192.168.6.0/24     10.6.0.1           UGS             wg0
192.168.8.0/24     192.168.60.1       UGS            igb2
192.168.8.8        192.168.60.1       UGHS           igb2
192.168.60.0/24    link#17            U            vlan06
192.168.60.6       link#6             UHS             lo0
192.168.60.166     link#6             UHS             lo0
**********      10.14.0.3          UGHS            wg2
**********./23    link#4             U              igb3
**********.     link#6             UHS             lo0
root@OC-OPNsense01:~ #

also while the condition persists:

root@OC-OPNsense01:~ # tcpdump -ni wg4 port 53
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wg4, link-type NULL (BSD loopback), snapshot length 262144 bytes
15:54:37.102597 IP 10.127.127.2.49847 > 10.127.127.1.53: 2782+ A? app-analytics-services.com. (44)
15:54:37.115132 IP 10.127.127.1.53 > 10.127.127.2.49847: 2782 6/0/0 A 192.179.26.138, A 192.179.26.139, A 192.179.26.100, A 192.179.26.102, A 192.179.26.101, A 192.179.26.113 (140)
15:54:37.176938 IP 10.127.127.2.57407 > 10.127.127.1.53: 8073+ A? mtalk.google.com. (34)
15:54:37.178119 IP 10.127.127.1.53 > 10.127.127.2.57407: 8073 2/0/0 CNAME mobile-gtalk.l.google.com., A 142.250.99.188 (79)
15:54:37.181977 IP 10.127.127.2.59601 > 10.127.127.1.53: 31715+ HTTPS? oauthaccountmanager.googleapis.com. (52)
15:54:37.187173 IP 10.127.127.2.61051 > 10.127.127.1.53: 3209+ A? oauthaccountmanager.googleapis.com. (52)
15:54:37.188306 IP 10.127.127.1.53 > 10.127.127.2.61051: 3209 8/0/0 A 172.217.114.4, A 172.217.119.4, A 172.217.116.4, A 172.217.112.4, A 172.217.117.4, A 172.217.113.4, A 172.217.118.4, A 172.217.115.4 (180)
15:54:37.194013 IP 10.127.127.1.53 > 10.127.127.2.59601: 31715 0/1/0 (109)
15:54:40.701918 IP 10.127.127.2.54830 > 10.127.127.1.53: 53225+ HTTPS? whatismyipaddress.com. (39)
15:54:40.704719 IP 10.127.127.1.53 > 10.127.127.2.54830: 53225 1/0/0 HTTPS (112)
15:54:40.707118 IP 10.127.127.2.51708 > 10.127.127.1.53: 59126+ A? whatismyipaddress.com. (39)
15:54:40.709549 IP 10.127.127.1.53 > 10.127.127.2.51708: 59126 2/0/0 A 104.19.223.79, A 104.19.222.79 (71)
15:54:41.677629 IP 10.127.127.2.56885 > 10.127.127.1.53: 54150+ A? safebrowsing.google.com. (41)
15:54:41.712736 IP 10.127.127.1.53 > 10.127.127.2.56885: 54150 5/0/0 CNAME sb.l.google.com., A 142.251.121.91, A 142.251.121.93, A 142.251.121.190, A 142.251.121.136 (124)

and after restarting the inbound wireguard service assosciated with wg4:

root@OC-OPNsense01:~ # ifconfig wg4
wg4: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1420
        description: 127OPNSense_WG_Server (opt15)
        options=80000<LINKSTATE>
        inet 10.127.127.1 netmask 0xffffff00
        groups: wg wireguard
        nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
root@OC-OPNsense01:~ #

root@OC-OPNsense01:~ # wg show wg4
interface: wg4
  public key: ***********************************=
  private key: (hidden)
  listening port: 51***

peer: ***********************************==
  preshared key: (hidden)
  allowed ips: 10.127.127.5/32
  transfer: 0 B received, 1.30 KiB sent
  persistent keepalive: every 25 seconds

peer: ***********************************==
  preshared key: (hidden)
  allowed ips: 10.127.127.2/32
  transfer: 0 B received, 1.30 KiB sent
  persistent keepalive: every 25 seconds

peer: ***********************************=
  preshared key: (hidden)
  allowed ips: 10.127.127.4/32
  transfer: 0 B received, 1.30 KiB sent
  persistent keepalive: every 25 seconds

peer: ***********************************=
  preshared key: (hidden)
  allowed ips: 10.127.127.3/32
  transfer: 0 B received, 1.30 KiB sent
  persistent keepalive: every 25 seconds

peer: ***********************************=
  preshared key: (hidden)
  allowed ips: 10.127.127.6/32
  transfer: 0 B received, 1.30 KiB sent
  persistent keepalive: every 25 seconds

peer: ***********************************=
  preshared key: (hidden)
  allowed ips: 10.127.127.7/32
  transfer: 0 B received, 1.30 KiB sent
  persistent keepalive: every 25 seconds

peer: ***********************************=
  preshared key: (hidden)
  allowed ips: 10.127.127.8/32
  transfer: 0 B received, 1.30 KiB sent
  persistent keepalive: every 25 seconds
root@OC-OPNsense01:~ #

root@OC-OPNsense01:~ # netstat -rn -f inet
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            **********.1       UGS            igb3
1.1.1.1            **********.1       UGHS           igb3
10.6.0.1           link#11            UHS             wg0
10.6.0.5           link#6             UH              lo0
10.10.112.0/24     link#5             U               re0
10.10.112.112      link#6             UHS             lo0
10.14.0.0/16       link#12            U               wg2
10.14.0.1          link#12            UHS             wg2
10.14.0.2          link#6             UHS             lo0
10.127.2.0/24      link#14            U            vlan02
10.127.2.1         link#6             UHS             lo0
10.127.3.0/24      link#15            U            vlan03
10.127.3.1         link#6             UHS             lo0
10.127.4.0/24      link#16            U            vlan04
10.127.4.1         link#6             UHS             lo0
10.127.7.0/24      link#18            U            vlan07
10.127.7.1         link#6             UHS             lo0
10.127.11.0/24     link#19            U           vlan011
10.127.11.1        link#6             UHS             lo0
10.127.12.0/24     link#20            U           vlan012
10.127.12.1        link#6             UHS             lo0
10.127.60.0/24     link#2             U              igb1
10.127.60.1        link#6             UHS             lo0
10.127.127.0/24    link#10            U               wg4
10.127.127.1       link#6             UHS             lo0
**********.      192.168.60.1       UGHS           igb2
127.0.0.1          link#6             UH              lo0
**********.     **********.       UGHS           igb3
**********.     **********.       UGHS           igb3
**********.    10.14.0.1          UGHS            wg2
192.168.6.0/24     10.6.0.1           UGS             wg0
192.168.8.0/24     192.168.60.1       UGS            igb2
192.168.8.8        192.168.60.1       UGHS           igb2
192.168.60.0/24    link#17            U            vlan06
192.168.60.6       link#6             UHS             lo0
192.168.60.166     link#6             UHS             lo0
**********.   10.14.0.3          UGHS            wg2
**********.    link#4             U              igb3
**********.    link#6             UHS             lo0
root@OC-OPNsense01:~ #

root@OC-OPNsense01:~ # tcpdump -ni wg4 port 53
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wg4, link-type NULL (BSD loopback), snapshot length 262144 bytes
16:00:36.087124 IP 10.127.127.2.64118 > 10.127.127.1.53: 5792+ A? clientservices.googleapis.com. (47)
16:00:36.087366 IP 10.127.127.1.53 > 10.127.127.2.64118: 5792 1/0/0 A 142.251.45.142 (63)
16:00:36.131512 IP 10.127.127.2.57188 > 10.127.127.1.53: 37694+ HTTPS? oauth2.googleapis.com. (39)
16:00:36.131542 IP 10.127.127.2.59348 > 10.127.127.1.53: 17623+ A? oauth2.googleapis.com. (39)
16:00:36.132579 IP 10.127.127.2.57722 > 10.127.127.1.53: 22940+ HTTPS? oauthaccountmanager.googleapis.com. (52)
16:00:36.143661 IP 10.127.127.1.53 > 10.127.127.2.59348: 17623 1/0/0 A 173.194.43.95 (55)
16:00:36.155291 IP 10.127.127.1.53 > 10.127.127.2.57188: 37694 0/1/0 (96)
16:00:36.169684 IP 10.127.127.1.53 > 10.127.127.2.57722: 22940 0/1/0 (109)
16:00:36.297808 IP 10.127.127.2.56151 > 10.127.127.1.53: 28843+ HTTPS? whatismyipaddress.com. (39)
16:00:36.301613 IP 10.127.127.2.49337 > 10.127.127.1.53: 12610+ A? whatismyipaddress.com. (39)
16:00:36.302416 IP 10.127.127.1.53 > 10.127.127.2.56151: 28843 1/0/0 HTTPS (112)
16:00:36.304267 IP 10.127.127.1.53 > 10.127.127.2.49337: 12610 2/0/0 A 104.19.223.79, A 104.19.222.79 (71)
16:00:36.402933 IP 10.127.127.2.49214 > 10.127.127.1.53: 3553+ A? mtalk.google.com. (34)
16:00:36.416478 IP 10.127.127.2.49813 > 10.127.127.1.53: 7412+ HTTPS? accounts.google.com. (37)
16:00:36.421684 IP 10.127.127.2.65452 > 10.127.127.1.53: 1891+ A? accounts.google.com. (37)
16:00:36.422557 IP 10.127.127.2.54808 > 10.127.127.1.53: 57995+ A? android.clients.google.com. (44)
16:00:36.422611 IP 10.127.127.2.61247 > 10.127.127.1.53: 44304+ A? optimizationguide-pa.googleapis.com. (53)
16:00:36.433098 IP 10.127.127.1.53 > 10.127.127.2.61247: 44304 8/0/0 A 172.217.115.4, A 172.217.114.4, A 172.217.112.4, A 172.217.119.4, A 172.217.116.4, A 172.217.118.4, A 172.217.117.4, A 172.217.113.4 (181)
16:00:36.433182 IP 10.127.127.1.53 > 10.127.127.2.65452: 1891 1/0/0 A 192.179.26.84 (53)
16:00:36.437620 IP 10.127.127.2.63627 > 10.127.127.1.53: 14093+ A? safebrowsing.google.com. (41)
16:00:36.463892 IP 10.127.127.1.53 > 10.127.127.2.49813: 7412 0/1/0 (87)
16:00:36.484631 IP 10.127.127.1.53 > 10.127.127.2.49214: 3553 2/0/0 CNAME mobile-gtalk.l.google.com., A 192.178.163.188 (79)
16:00:36.495860 IP 10.127.127.1.53 > 10.127.127.2.63627: 14093 5/0/0 CNAME sb.l.google.com., A 172.253.117.136, A 172.253.117.91, A 172.253.117.93, A 172.253.117.190 (124)
16:00:36.504379 IP 10.127.127.1.53 > 10.127.127.2.54808: 57995 17/0/0 CNAME android.l.google.com., A 173.194.203.113, A 192.178.163.101, A 192.178.163.139, A 173.194.43.102, A 173.194.43.138, A 173.194.43.113, A 173.194.43.101, A 173.194.43.139, A 173.194.43.100, A 173.194.202.113, A 173.194.202.102, A 173.194.203.100, A 173.194.203.139, A 173.194.203.101, A 173.194.203.138, A 173.194.203.102 (324)
16:00:36.557426 IP 10.127.127.2.57793 > 10.127.127.1.53: 4819+ A? map1.whatismyipaddress.com. (44)
16:00:36.557442 IP 10.127.127.2.53819 > 10.127.127.1.53: 21823+ A? cmp.inmobi.com. (32)
16:00:36.557448 IP 10.127.127.2.61382 > 10.127.127.1.53: 24745+ A? a.pub.network. (31)
16:00:36.557454 IP 10.127.127.2.62172 > 10.127.127.1.53: 47220+ A? www.googletagmanager.com. (42)
16:00:36.557459 IP 10.127.127.2.62954 > 10.127.127.1.53: 13340+ HTTPS? ds6.probe.whatismyipaddress.com. (49)
16:00:36.559827 IP 10.127.127.1.53 > 10.127.127.2.57793: 4819 2/0/0 A 104.19.222.79, A 104.19.223.79 (76)
16:00:36.561446 IP 10.127.127.2.54688 > 10.127.127.1.53: 32486+ A? ds6.probe.whatismyipaddress.com. (49)
16:00:36.565763 IP 10.127.127.1.53 > 10.127.127.2.62954: 13340 0/1/0 (108)
16:00:36.567314 IP 10.127.127.1.53 > 10.127.127.2.54688: 32486 0/1/0 (108)
16:00:36.577499 IP 10.127.127.2.55008 > 10.127.127.1.53: 54460+ HTTPS? a.pub.network. (31)
16:00:36.581438 IP 10.127.127.2.63439 > 10.127.127.1.53: 16517+ HTTPS? app.fusebox.fm. (32)

NOTE: I anonymized some of the public IPs, ports, and keys. If any of that is necessary for troubleshooting, please let me know and I'll provide it.

Thanks again for looking at this!

Hello meyergru,

Just so I know which info to collect for not confusing you - which interface should that be? As I think we now ruled out the initial suspect (inbound WG). Just for recollection - the OPNsense boots normally with the three outbound WG connections to names instead of IPs disabled, and with the inbound connection enabled. Hence it's one or more of the three causing the unwanted behavior (pointed out by others as well).

Would it be simpler to disable two of the three outbound WG connections, and provide the output of the third one? At least that way there would be some clear attribution...Thank you!

Hello Gilgamesh,

I have the same issue. Sharing with you my topic: https://forum.opnsense.org/index.php?topic=52901.0

Did you find any solution in the meantime?

seems like I'm not the only one - 26.7: https://forum.opnsense.org/index.php?topic=52711.0

and a much older entry: https://forum.opnsense.org/index.php?topic=37223.0

I'll provide the requested diagnostic information shortly. Thank you!

And another troubleshooting attempt and its result: I have temporarily replaced the three outbound tunnel endpoints DNS names with their external IPs, and rebooted. In this configuration, the OPNsense rebooted quickly, and the inbound WG client DNS resolution works flawlessly.

Therefore it appears that the online advice (of replacing names with IPs for WG endpoints) improves things, but this isn't a workable solution, as some endpoints' IPs will change over time.

Is there any way to influence the ordering of the start of the OPNsense services? Ideally, wireguard would only start after DNS (both Unbound and DNSmasq) is fully up and running - as mentioned previously, I notice two instances of Unbound starting, one before and one after the "Configurating Wireguard VPN"...

Apologies for the later reply - in my timezone folks were working and I couldn't reboot the OPN during the day.

Further troubleshooting steps:

1. I have disabled all the wireguard peers and rebooted the OPNsense. Boot time was fast, similar to before enabling WG on it.

2. I have left disabled all the wireguard outbound peers (3 of them, two to public VPN providers, and one to a personal VPN - going to three different countries), but re-activated all the inbound VPN peers (the OPN as server and the various clients such as my phone, tablet, laptop, etc.) and rebooted the OPNsense. Boot time was also fast, similar to before enabling WG on it. Also - interestingly - the DNS resolution worked on the client I've tested it on (my mobile phone, connected to the OPNsense via cellular data). Note: I've only rebooted once in this configuration, so perhaps the fact that the client I tested with could resolve was a fluke, but I somehow doubt it.

3. I have then (after step 2, without re-rebooting the OPNsense) reactivated the three outbound peers, but they were flakey. I tried to refresh the individual instances in the dashboard, but the flakiness persisted. However, upon refreshing the wireguard service itself all re-became stable and all "lights are green".

4. Some observations: when booting, I noticed the following sequence:

....
....
Starting DNSmasq
Starting Unbound DNS
....
....
Configuring Wireguard VPN
....
....
Starting Unbound DNS
....
....

Q1: Is it normal that Unbound DNS appears twice, before and after the Wireguard VPN service start?

Q2: These observations point to the opinions expressed online, that the issues are when having dynamic IPs resolved through name resolution as outbound VPN connections. I thought some more about that, and the issue is that indeed the TTL on those records are short, between 60 and 300 seconds. I don't have control over those, but I have created CNAMEs pointing to them with 3600s TTLs and used those as peer names - but that doesn't help either, as (I believe) Unbound is clearing its cache upon reboot, and therefore a new query will still be necessary if the remote peers are addressed by name upon reboot, which brings us back to the Unbound vs. WG chicken-or-egg conundrum. Any workable solution? Thank you, and have a great weekend!

[do not bind any services to specific interfaces]

Danke sehr, meyergru!

Well, now I got you. I suspect this might be one of two cases:

1. You have bound your DNSmasq or Unbound to specific interfaces. In that case, when Unbound starts up, the WG interface does not yet exist and cannot be bound. You can test if this is the case by "sockstat -4 -6 -l | grep :53" and looking at which interfaces WG listens on when your error condition exists. If that is the case, you can avoid it by using "ALL" (nor no) interfaces to have the DNS service bind to 0.0.0.0. This is explained often throughout the forum: do not bind any services to specific interfaces. Besides, this is here: https://forum.opnsense.org/index.php?topic=42985.0, point 32.

I have confirmed that both DNSmask and Unbound are listening to all interfaces, and that Unbound has the Default action: Allow. Nevertheless, there's an ACL entry, but that one should be rendered moot by the "default allow", supposedly. I have also tried disabling that ACL, confirmed that Unbound was still resolving (which makes sense, given the 'Allow' default action) - but even with the ACL disabled, "Configuring Wireguard" took minutes when rebooting, and resulted in WG clients unable to resolved until the WG instance was restarted.

2. If you are using Unbound, there is another known possibility: the automatically generated ACL for the WireGuard network may be missing after boot because the WG interface does not yet exist when the Unbound configuration is generated.

This has been reported before:
https://github.com/opnsense/core/issues/4142

Check /var/unbound/access_lists.conf while the problem exists and see whether your WG subnet is present. If it only appears after restarting Unbound, that is the cause.

Unlike the interface binding problem, the simple workaround here is to add an explicit ACL for the WireGuard client network under Services -> Unbound DNS -> Access Lists instead of relying on the automatically generated interface ACL.

This second case should only matter if your Unbound ACL default action is set to Deny or Refuse. With the default action set to Allow, the automatically generated interface ACL should be irrelevant (see: https://forum.opnsense.org/index.php?topic=42985.0, point 0). The old ticket was created when there where specific interface rules created (namely, in 2020).

Would all this section apply, since I do NOT have to restart Unbound to make it all work? As mentioned above, I have even tried restarting Unbound, but this did NOT fix the resolution issues for WG clients - the only thing that does is restarting that WG inbound instance.

What else I've tried but did not fix the issue:

1. I've temporarily disabled the  "Renew DNS for Wireguard on stale connections" cron job.
2. I've temporarily replaced the short-ttl (1-5min) dynamic IP DNS entries with 60min CNAMEs pointing to them.

I somehow tie it all to the inbound Wireguard instance. I do not remember having had that issue before enabling inbound Wireguard. But, like stated, once I restart that instance after the OPNsense has (eventually) rebooted, it works perfectly until the next restart.

My understanding is that the "Renew DNS for Wireguard on stale connections" script helps when one of the S2S endpoints with a dynamic IP resolved by means of an alias is changing its IP *while the tunnel is up* (it has nothing to do with a reboot). On other devices wireguard is notorious to fail to re-establish a connection if the dynamic IP at the other end is changing while the tunnel is established (and manual intervention is required - disabling and re-enabling the tunnel brings it back right away). For instance, on Windows I've written a PowerShell script that I ran every 5 minutes via Scheduled Tasks - if the remote gateway was unavailable over ICMP it restarted the tunnel, which reliably re-established the connection. For instance, on GL.iNET devices the watchguard feature is well implemented, it never led to any issues.

But here we're not talking about that watchdog feature, but instead by the OPNsense wireguard server being unable to resolve its clients' DNS queries after rebooting the firewall, and until the service is restarted manually (I say again, S2S connections work well, it's only about clients which connect but cannot resolve). It has nothing to do with client-side dynamic IPs. According to what I found online, it's because the OPNsense wg 'server' service (instance) attempts to run before the DNS resolver is fully up - and the suggestion solution is scripting the wg server to start after a certain delay (they are proposing 30 seconds, which I can live). But the reason I've written here is because I'd like a Devs-endorsed workaround, rather than something I've found online - especially since it's likely that it would help many others as well.