Messages

[So the question ist, did I do something wrong or is there a Bug? (or both)]

There is a very long story here, I'll start with the TL;DR.

Syncing from Master -> Slave almost doubled the .XML file on the slave
Going from 100k on the master to 24M on the slave.
Eventually loading/using the config rendered the slave unusable.

Looking at the file, it seems every SYNC doubled the entrys for CARP-VIPs that are set to "nosync"
In an attempt to reduce multicast, i went with an unicast peer for carp
Syncing those seemed not reasonable, because the peer IP would always stay the same.
So i configured both Firewall in the same same way syncing would (increasing advskew)

Additional IPs are virtual IPs with the same vhid.
Synced IPALIAS VIPs are not doubled.

GUI shows only one instance, makes sense since the uuid als always the same.
Everything is working as expected, until the config file gets to large.

Not sure where to go, multicast would probably work, but I would very much like to remove multicast traffic where ever possible.

So the question ist, did I do something wrong or is there a Bug? (or both)

I have 32 identical entrys of this:

<vip uuid="76b1ef2d-39bc-4aef-9999-a208a9e29bba">
      <interface>opt13</interface>
      <mode>carp</mode>
      <subnet>10.4.171.251</subnet>
      <subnet_bits>24</subnet_bits>
      <gateway/>
      <noexpand>0</noexpand>
      <nobind>0</nobind>
      <password>10.4.171.251/24</password>
      <vhid>171</vhid>
      <advbase>1</advbase>
      <advskew>100</advskew>
      <peer>10.4.171.252</peer>
      <peer6/>
      <nosync>1</nosync>
      <descr/>
    </vip>

IPALIAS not doubled:

<vip uuid="4652b53b-0d82-43cb-899e-2a663e7a47c2">
      <interface>opt13</interface>
      <mode>ipalias</mode>
      <subnet>10.4.171.1</subnet>
      <subnet_bits>24</subnet_bits>
      <gateway/>
      <noexpand>0</noexpand>
      <nobind>0</nobind>
      <password>10.4.171.251/24</password>
      <vhid>171</vhid>
      <advbase>1</advbase>
      <advskew>100</advskew>
      <peer>10.4.171.253</peer>
      <peer6/>
      <nosync>0</nosync>
      <descr/>
    </vip>

Long Story:
I had to temporarily use a Backup Firewall for an emergency.
I created a Backup, reconfigered the device.
Later, I restored the Backup and everything was fine until syncing changes from the primary firewall.

- A lot of php-scripts were at 100% CPU rendering the webserver unusable
- Rebooting took 30-60 minutes... (slowed down when the config was read)
- It started after syncing changes from the master to the backup
- only solution was to restore an old version through console/ssh
- issues came back after syncing again (and only then)
- I tried to compare versions through the GUI... CLIENT-CPU went up to 100% for the browser pocess.

According to the GUI there was 8.6M used for all local config backups.

Code Select Expand

Be aware of how much space is consumed by backups before adjusting this value. Current space used: 8.6MThis is what it actually looked like:

Code Select Expand

root@[...]-RT-01-B:~ # ls -lh /conf/backup/
total 8814
-rw-r-----  1 root wheel   90K May 27 08:56 config-1748328980.507.xml
-rw-r-----  1 root wheel  128K May 27 08:58 config-1748329134.0709.xml
-rw-r-----  1 root wheel  173K May 27 08:59 config-1748329199.4194.xml
-rw-r-----  1 root wheel  173K May 27 09:14 config-1748330077.7664.xml
-rw-r-----  1 root wheel  262K May 27 09:17 config-1748330233.9693.xml
-rw-r-----  1 root wheel  440K May 27 09:20 config-1748330444.7983.xml
-rw-r-----  1 root wheel  440K May 27 09:21 config-1748330474.8112.xml
-rw-r-----  1 root wheel  799K May 27 09:21 config-1748330505.4913.xml
-rw-r-----  1 root wheel  1.5M May 27 09:22 config-1748330572.1054.xml
-rw-r-----  1 root wheel  2.9M May 27 09:27 config-1748330860.5138.xml
-rw-r-----  1 root wheel  2.9M May 27 09:29 config-1748330964.7592.xml
-rw-r-----  1 root wheel  2.9M May 27 09:35 config-1748331343.0051.xml
-rw-r-----  1 root wheel   88K May 27 09:36 config-1748331386.3617.xml
-rw-r-----  1 root wheel   88K May 27 09:46 config-1748331972.5229.xml
-rw-r-----  1 root wheel   94K May 27 09:48 config-1748332101.2589.xml
-rw-r-----  1 root wheel  105K May 27 09:54 config-1748332499.61.xml
-rw-r-----  1 root wheel  127K May 27 09:56 config-1748332570.9768.xml
-rw-r-----  1 root wheel  127K May 27 09:57 config-1748332642.9083.xml
-rw-r-----  1 root wheel  127K May 27 09:58 config-1748332725.239.xml
-rw-r-----  1 root wheel  172K May 27 10:00 config-1748332802.6074.xml
-rw-r-----  1 root wheel  261K May 27 12:07 config-1748340470.8171.xml
-rw-r-----  1 root wheel  260K May 27 12:08 config-1748340509.1095.xml
-rw-r-----  1 root wheel  260K May 27 12:14 config-1748340855.321.xml
-rw-r-----  1 root wheel  441K May 27 13:32 config-1748345544.0094.xml
-rw-r-----  1 root wheel  440K May 27 13:32 config-1748345550.9518.xml
-rw-r-----  1 root wheel  798K May 27 13:34 config-1748345640.3239.xml
-rw-r-----  1 root wheel  1.5M May 27 13:49 config-1748346540.7216.xml
-rw-r-----  1 root wheel  2.9M May 27 13:50 config-1748346612.6832.xml
-rw-r-----  1 root wheel  5.7M May 27 13:58 config-1748347103.7655.xml
-rw-r-----  1 root wheel   11M May 27 14:03 config-1748347396.8214.xml
-rw-r-----  1 root wheel   95K May 27 14:11 config-1748347896.6312.xml
-rw-r-----  1 root wheel  102K May 27 14:11 config-1748347912.9957.xml
-rw-r-----  1 root wheel  113K May 27 14:13 config-1748347980.5211.xml
-rw-r-----  1 root wheel  138K May 27 14:28 config-1748348926.8479.xml
-rw-r-----  1 root wheel  185K May 27 14:39 config-1748349561.5601.xml
-rw-r-----  1 root wheel  274K May 27 14:42 config-1748349724.5685.xml
-rw-r-----  1 root wheel  274K May 27 14:49 config-1748350176.9072.xml
-rw-r-----  1 root wheel  274K May 27 14:50 config-1748350250.0818.xml
-rw-r-----  1 root wheel  274K May 27 14:51 config-1748350296.2211.xml
-rw-r-----  1 root wheel  275K May 27 14:52 config-1748350329.1173.xml
-rw-r-----  1 root wheel  455K May 27 14:52 config-1748350376.2913.xml
-rw-r-----  1 root wheel  102K May 27 15:11 config-1748351470.2933.xml
-rw-r-----  1 root wheel  109K May 27 15:12 config-1748351531.7346.xml
-rw-r-----  1 root wheel  122K May 28 08:44 config-1748414661.7584.xml
-rw-r-----  1 root wheel  170K May 30 09:19 config-1748589587.9938.xml
-rw-r-----  1 root wheel  170K May 30 09:21 config-1748589682.036.xml
-rw-r-----  1 root wheel  170K May 30 09:21 config-1748589687.3124.xml
-rw-r-----  1 root wheel  170K May 30 09:21 config-1748589692.7848.xml
-rw-r-----  1 root wheel  170K May 30 09:21 config-1748589695.892.xml
-rw-r-----  1 root wheel  221K May 30 09:25 config-1748589920.9901.xml
-rw-r-----  1 root wheel  319K May 30 09:26 config-1748590014.7545.xml
-rw-r-----  1 root wheel  514K May 30 09:32 config-1748590373.9919.xml
-rw-r-----  1 root wheel  909K May 30 09:58 config-1748591926.6014.xml
-rw-r-----  1 root wheel  1.7M May 30 10:05 config-1748592355.8058.xml
-rw-r-----  1 root wheel  3.2M May 30 10:55 config-1748595311.5473.xml
-rw-r-----  1 root wheel  3.2M May 30 10:55 config-1748595330.9273.xml
-rw-r-----  1 root wheel  3.2M May 30 10:55 config-1748595349.6193.xml
-rw-r-----  1 root wheel  3.2M May 30 10:56 config-1748595361.0046.xml
-rw-r-----  1 root wheel  6.2M May 30 11:04 config-1748595878.2988.xml
-rw-r-----  1 root wheel  6.2M May 30 11:14 config-1748596451.7341.xml
-rw-r-----  1 root wheel  6.2M May 30 11:15 config-1748596537.4666.xml
-rw-r-----  1 root wheel  6.2M May 30 11:16 config-1748596575.0691.xml
-rw-r-----  1 root wheel  6.2M May 30 11:19 config-1748596777.155.xml
-rw-r-----  1 root wheel   12M May 30 11:25 config-1748597101.168.xml
-rw-r-----  1 root wheel   24M May 30 11:48 config-1748598536.491.xml
-rw-r-----  1 root wheel   24M May 30 11:53 config-1748598793.3228.xml
-rw-r-----  1 root wheel   24M May 30 12:41 config-1748601677.3911.xml
-rw-r-----  1 root wheel  170K Jun  3 09:57 config-1748937465.8225.xml


Last Update.

Problem lies within the ACME Account-Config, in my case LetsEncrypt.
acme.sh saves credantials it needs for validation under /var/etc/acme-client/accounts/<ACME-Account>/account.conf

It had a "SAVED_AZUREDNS_BEARERTOKEN", that was old and "SAVED_AZUREDNS_TOKENVALIDTO", that was 0.
So it used the invalid token without even checking.

Removing those lines fixed the issue (at least on 24.7.12_4)

New Config contains an emtpy "SAVED_AZUREDNS_BEARERTOKEN", a valid "SAVED_AZUREDNS_ACCESSTOKEN" and a proper "SAVED_AZUREDNS_TOKENVALIDTO".
Creating a new Account should have fixed the issue as well.

As a side note, for FreeDNS acme.sh. puts a cookie in that file.
That also stopped working eventually.

Probably an Edge-case, depending on the verion we set AzureDNS up.
OPNsense versions with acme.sh 3.10 should be fine for new setups.

not so quick and very dirty Workaround:

1) log into Azure using poswershell with App-IP/Secret/Tennant-ID thats used in the Plugin

Code Select Expand

az login --service-principal -u <AppID> -p <AppSecret> --tenant <TenantID>2) get the current AuthToken

Code Select Expand

az account get-access-token3) add the Token as an export

Code Select Expand

vi /usr/local/opnsense/mvc/app/library/OPNsense/AcmeClient/LeValidation/DnsAzure.php
add $this->acme_env['AZUREDNS_BEARERTOKEN'] = "String from 2)"4) rerun Validation within 1 Hour (see "expiresOn" in 2)

5) hope for a fix or look for another validation Method

If I read this right, bearer token should only be used if it exists:
https://github.com/acmesh-official/acme.sh/blob/master/dnsapi/dns_azure.sh

I think that change is making issues:

https://github.com/acmesh-official/acme.sh/pull/5276

Same issue here.
It broke around early November.

Same User/Token is used on multiple OPNsense and pfSense installations.
All OPNsense stopped working around November, all pfSense still work today.
Manually using the Azure-API through Powershell with the same User/Token also works.

Following log Entry is probably the Key here:

Code Select Expand

[Thu Jan 16 12:23:59 CET 2025] response {"error":{"code":"ExpiredAuthenticationToken","message":"The access token expiry UTC time '11/1/2024 3:15:32 AM' is earlier than current UTC time '1/16/2025 11:23:59 AM'."}}='[hidden](please add '--output-insecure' to see this value)'The original Key is valid until 07/2025
I created a new Key, (valid until 01/2027) but the Error did not change.

I think the "AuthenticationToken" is not the Key, that is created in Azure and configured in the Plugin, but a token that should be renewed when authentication is succsessfull.

I could be wrong.

There were some Updates to AzureDNS in November.
https://azure.microsoft.com/en-us/updates?filters=%5B%22Azure+DNS%22%5D