Messages

I feel like you ran into this here:
https://github.com/opnsense/plugins/pull/5102

Sounds like an external Bind server might be better, I don't know if this can be reasonably fixed in the plugin.
If your workarounds work, well, that's good though, nice job figuring this all out.

Thank you! This is indeed what I am bumping into. I wonder if it is possible to write DDNS changes into a separate zone file. That way, the journal can monitor that file instead.

OP, if you ever decide to go down this path, I suggest to first set up Monit to watch log files and run a script based on keywords. When you see journal rollforward failed, that means Bind did not load that zone file. Your static DNS assignments in addition to Kea DDNS registrations are all offline at that point. Internet through Unbound still works but no LAN devices are resolvable. You need Monit to watch for that, quickly delete those journal files and restart Bind to recover before anyone notices. Until there is a better way, that is how I have it set up. I actually match on zone not loaded as my keyword.

I felt brave and decided to bite the bullet. After about 12 hours, I didn't get it all working so I caught a little hell from my wife. 😬 I think I finally got most of the fires out so below are my notes:

• I decided to break out my different VLANs (e.g. IoT) into their own forward zones. I used sub-zones in the format vlan.domain.tld. This method makes it easier to troubleshoot.
• Uncheck Register ISC DHCP4 Leases and Register DHCP Static Mappings under Services > Unbound > General to avoid creating entries in Unbound since I still have ISC dhcpd installed.
• Restarting Bind caused the zone reload to fail with journal rollforward failed: journal out of sync with zone errors in the Bind log. I used rndc sync -clean to clear out the *.jnl files before restarting. I need to find a better way since I lose DDNS entries without the journal. I tried rndc stop and rndc freeze before clicking Save and neither fixed it.
• Under each Kea subnet DDNS setting, I had to end the forward and reverse zones, and qualifying suffix fields with a dot (.). Otherwise, Bind could not find the correct zone to update.
• I got DDNS update failures with 'RRset exists (value dependent)' errors in Bind for devices that are IPv4+IPv6 dual stack. I think this is due to both using the same DHCID and it became a race to create the first record. I set no-check-with-dhcid in each Kea subnet to fix it.
• My gateway was named "gw.vlan.domain.tld". Even when Unbound is set to forward all vlan.domain.tld queries to Bind, it refused to forward this one. I had to change my gateway domain name under System > Settings > General > Domain to something else.
• Some devices (OpenWRT) only send host names to Kea. This is what "DNS qualifying suffix" was supposed to fix. That setting worked for IPv4 but not IPv6; maybe OpenWRT sent FQDNs on IPv4 but I did not check. Kea logs showed No DNS servers match FQDN hostname. Reserving their IPv6 address fixed it. I think this issue is with upstream but that requires confirmation. Any IPv6 lease that is not FQDN under the Hostname column needs this reservation.
• I used dig @x.x.x.x -p 53530 name to query Bind directly. Otherwise, I got Unbound's cached entry.
• Once everything is confirmed to work, I set Bind to listen on 127.0.0.1 and ::1 so networks cannot query it directly.

So far so good. I had to create some CNAME records for our printer and SIP proxy. This gets Windows and desk phone working until I get time to reconfigure them and switch to using their IoT names instead.

Good point. If I am considering a workaround, I should at least look at other options as well. Thanks for that suggestion.

But the Unbound DNS registration for dynamic hosts will most likely not come, only DHCP reservations are synced with Unbound -> which should actually be sufficient for most setups.

I saw a discussion where someone got a script to do that. It is likely the direction I will go, but I figure I'll ride this ISC horse until the sun sets. :)

Prefix delegation and dhcp registration into Unbound is why I am still running the ISC dhcpd. Aside from my home lab, Apple's HomeKit uses prefix delegated IPv6 addresses. I guess it assigns IPs to downstream nodes.

The following expressions include IPv4 and IPv6 addresses.

• All Regions:
  
  Code Select Expand
  .prefixes,.ipv6_prefixes|map(.ip_prefix//.ipv6_prefix)[]
• GLOBAL and US regions:
  
  Code Select Expand
  .prefixes,.ipv6_prefixes|map(select(.region=="GLOBAL" or (.region|startswith("us-")))|.ip_prefix//.ipv6_prefix)[]

Edit:

Use this to list all regions.

Code Select Expand

jq '.prefixes+.ipv6_prefixes|map(.region)|unique' ip-ranges.json

[ scratch that. I should not post out of frustration especially when I am unable to gather more info to help troubleshoot. ]

IPv6-PD is not commonly used and it is not actively monitored-at least by Tier 1 support since they told me their diagnostics all show green.

If that's the case for business accounts... then the fact that IPv6-PD works at all for my home connection is something of a miracle and I'm on my own.

Great.

I have no evidence of this, but I am guessing business and residential accounts all go thru the same support structure. We just get a different modem and our techs wear shirts and drive trucks that say Comcast Business. We also had AT&T's different broadband offerings going back to DSL in the 90s and we had similar experiences there as well. None of them had a way for technically savvy customers to help them troubleshoot. DSL Reports forums were a lifeline back then.

If you are affected by this, you'll want to get a ticket opened and request a firmware rollback.

I really want to emphasize this to anyone reading. IPv6-PD is not commonly used and it is not actively monitored-at least by Tier 1 support since they told me their diagnostics all show green. It takes everyone affected to call and open a ticket before someone notices. The call volume has to be enough to show up in their reports. Otherwise, our issue stays below their radar and they consider us "isolated issues".

Thanks for telling me about this thread, Franco. I spoke to someone in their corporate escalations group on Nov 10. Even he had to find a way to get it escalated into their engineering group. By Dec 1st, they rolled back my firmware at my request and I confirmed that fixed the problem (again). They then started rolling everyone back on Dec 5th and expected to complete that process by Dec 8th. He was going to update me if things change and I was going to reach out if the rollback caused issues. Thankfully, all went well.

Sadly, this is not the first time firmware updates affected my IPv6. My previous event triggered the modem's firewall and *block all incoming IPv6 connections* even though it is set to "disabled". Port forwarding, IPSec, client VPNs all went down. Similar to this time, I found someone who was able to relay it into engineering.

Btw, the one I am eagerly awaiting news on is the CheckPoint vs StrongSwan 6.0.3 CHILD_CREATE issue we had (#9382). The latest info I received today was their R&D discussed my case in their meeting and they will investigate my issue before making a decision. I set up a lab to gather logs and sent it all in along with Tobias' comments and links to the RFCs. I hope it was convincing enough.

OPNsense can also do GPS, but I know of nobody using that.

I went through an NTP+GPS phase and I had this GPS connected to an RS232 port running bare metal OPNsense. There is a tunable to pull PPS from DCD and I got its offset down to below 10ns. Sadly, I had to switch to USB when I upgraded and my offset is now regularly in the 0.1-0.5ms range.

OPNsense provides NTP throughout the house and I have 2 other units on the local network for accurate NTP sync.

Code Select Expand

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
o127.127.20.0    .GPS.            0 l   10   16  377    0.000   -0.168   0.260
 0.opnsense.pool .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 1.opnsense.pool .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 2.opnsense.pool .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 3.opnsense.pool .POOL.          16 p    -   64    0    0.000   +0.000   0.000
+2603:3018:143b: .PPS.            1 u    9   16  377    0.563   +0.159   0.085
+192.168.1.124   192.168.1.10     2 s    8   16  377    0.262   +0.155   0.033
-23.150.40.242   204.9.54.119     2 u   37   64  377   37.326   +3.317   4.652
-2603:c020:0:836 132.163.97.4     2 u   15   64  377   68.738   +3.902   1.751
-158.51.99.19    204.9.54.119     2 u   27   64  377   33.253   +8.254   1.336
-2606:82c0:23::e 216.239.35.0     2 u   37   64  377   34.156   +4.076   2.105
-15.204.246.57   94.0.219.24      2 u   10   64  377   36.513   +3.360   1.836
+144.202.0.197   207.66.79.103    2 u   25   64  377   34.933   +3.891   1.411


[Reset ACME Client]

My remaining certificates renewed this morning. Under "Services > ACME Client > Log Files > System Log tab", do you see a non-zero value for "AcmeClient: AcmeClient: The shell command returned exit code 'n'"? Are you able to cat out the file at the end of that line? On my error post above, it is /var/etc/acme-client/accounts/6027ee9e097f39.62139316_prod/account.conf. Until I hit Reset ACME Client, that file did not exist. You can also try increasing the ACME logging level from "normal" to "debug" before the next renewal.

[Reset ACME Client]

Further searches through this forum produced the following links:

• https://forum.opnsense.org/index.php?topic=30249.msg147778#msg147778
• https://github.com/opnsense/plugins/issues/3154

The scenario made sense as I recently migrated to new hardware and imported config.xml. The recommendation is to click on Reset ACME Client. I was presented the following (emphasis mine) and I am confident this is the solution.

This will remove ALL certificates, private keys, CSRs from ACME Client and reset all certificate and account states. However, existing certificates will remain in OPNsense trust storage. The ACME Client will automatically regenerate everything on its next scheduled run. This is most useful when importing a config backup to a new firewall. Continue?

I notice I have the same problem on 25.7.2. All 3 of my certificates failed to renew but it works when I manually click on the button to "Issue or renew certificate". Acme-client logs show the error: 'host.domain.tld' is not an issued domain, skipping.

System log for the failure says:
AcmeClient: The shell command returned exit code '2': '/usr/local/sbin/acme.sh --renew --syslog 6 --log-level 1 --serv
er 'letsencrypt' --webroot /var/etc/acme-client/challenges --home '/var/etc/acme-client/home' --cert-home '/var/etc/acme-client/cert-home/6027eeb80684e4.42843464' --certpath '/var/etc/acme-client/certs/6027eeb80684e4.4
2843464/cert.pem' --keypath '/var/etc/acme-client/keys/6027eeb80684e4.42843464/private.key' --capath '/var/etc/acme-client/certs/6027eeb80684e4.42843464/chain.pem' --fullchainpath '/var/etc/acme-client/certs/6027eeb806
84e4.42843464/fullchain.pem' --domain 'host.domain.tld' --days '60'   --keylength 'ec-384' --ecc --accountconf '/var/etc/acme-client/accounts/6027ee9e097f39.62139316_prod/account.conf''

System log for the successful says:
AcmeClient: The shell command returned exit code '0': '/usr/local/sbin/acme.sh --issue --syslog 6 --log-level 1 --server 'letsencrypt' --webroot /var/etc/acme-client/challenges --home '/var/etc/acme-client/home' --cert-home '/var/etc/acme-client/cert-home/6027eeb80684e4.42843464' --certpath '/var/etc/acme-client/certs/6027eeb80684e4.42843464/cert.pem' --keypath '/var/etc/acme-client/keys/6027eeb80684e4.42843464/private.key' --capath '/var/etc/acme-client/certs/6027eeb80684e4.42843464/chain.pem' --fullchainpath '/var/etc/acme-client/certs/6027eeb80684e4.42843464/fullchain.pem' --domain 'host.domain.tld' --days '60' --force  --keylength 'ec-384' --accountconf '/var/etc/acme-client/accounts/6027ee9e097f39.62139316_prod/account.conf''


I see 3 differences between the shell commands. Perhaps one of them is the difference between a successful and failed renewal.

• --renew (failed) vs --force
• --force (succeeds)
• --ecc (failed)