Messages

Moin,

ich habe diesen Post schon im Englischen Teil des Forums gepostet: https://forum.opnsense.org/index.php?topic=48478.0

Wir haben folgendes Problem. Seit 3 Tagen sind bei uns mehrmals die Firewalls kurz nacheinander gecrashed. Wir betreiben ein HA Pärchen mit der identischen Hardware. Dabei ist die Backup Firewall "nur" 2x gecrashed. Auszug aus dem crashreport zeigt auf ein "Panic String: page fault". Falls mehr Informationen benötigt werde, bitte bescheid geben oder wenn jemand eine Idee hat, wo wir nach dem Fehler suchen können.

Ein memtest haben wir bereits gestartet. Ergebnis noch offen.

Auszug aus /var/crash/info.* der Masterfirewall:

Code Select Expand

Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 155648
  Blocksize: 512
  Compression: none
  Dumptime: 2025-08-12 17:33:07 +0200
  Hostname: OPNsense1.localdomain
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 14.2-RELEASE-p4 stable/25.1-n269832-6addeda7db20 SMP
  Panic String: page fault
  Dump Parity: 3292242252
  Bounds: 0
  Dump Status: good

Code Select Expand

Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 116224
  Blocksize: 512
  Compression: none
  Dumptime: 2025-08-12 17:48:11 +0200
  Hostname: OPNsense1.localdomain
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 14.2-RELEASE-p4 stable/25.1-n269832-6addeda7db20 SMP
  Panic String: page fault
  Dump Parity: 1275568460
  Bounds: 1
  Dump Status: good

Code Select Expand

Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 155648
  Blocksize: 512
  Compression: none
  Dumptime: 2025-08-14 15:20:20 +0200
  Hostname: OPNsense1.localdomain
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 14.2-RELEASE-p4 stable/25.1-n269832-6addeda7db20 SMP
  Panic String: page fault
  Dump Parity: 3817318220
  Bounds: 2
  Dump Status: good
Auszug aus /var/crash/info.* der Backupfirewall:

Code Select Expand

Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 155648
  Blocksize: 512
  Compression: none
  Dumptime: 2025-08-12 22:40:23 +0200
  Hostname: OPNsense2.localdomain
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 14.2-RELEASE-p4 stable/25.1-n269832-6addeda7db20 SMP
  Panic String: page fault
  Dump Parity: 3234046287
  Bounds: 0
  Dump Status: good

Code Select Expand

Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 155648
  Blocksize: 512
  Compression: none
  Dumptime: 2025-08-14 15:37:13 +0200
  Hostname: OPNsense2.localdomain
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 14.2-RELEASE-p4 stable/25.1-n269832-6addeda7db20 SMP
  Panic String: page fault
  Dump Parity: 4270040911
  Bounds: 1
  Dump Status: good

Hello,

There have been several firewall crashes over the last 3 days. We operate an HA configuration with two identical hardware devices, both of which crashed in quick succession. The backup firewall only crashed twice instead of three times. If you need further information, please let us know what you need. Or if you have any idea where we can look for the cause.

A memtest is already running on the backup firewall to check for errors.


Here is the /var/crash/info.* from the Master Firewall:

Code Select Expand

Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 155648
  Blocksize: 512
  Compression: none
  Dumptime: 2025-08-12 17:33:07 +0200
  Hostname: OPNsense1.localdomain
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 14.2-RELEASE-p4 stable/25.1-n269832-6addeda7db20 SMP
  Panic String: page fault
  Dump Parity: 3292242252
  Bounds: 0
  Dump Status: good

Code Select Expand

Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 116224
  Blocksize: 512
  Compression: none
  Dumptime: 2025-08-12 17:48:11 +0200
  Hostname: OPNsense1.localdomain
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 14.2-RELEASE-p4 stable/25.1-n269832-6addeda7db20 SMP
  Panic String: page fault
  Dump Parity: 1275568460
  Bounds: 1
  Dump Status: good

Code Select Expand

Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 155648
  Blocksize: 512
  Compression: none
  Dumptime: 2025-08-14 15:20:20 +0200
  Hostname: OPNsense1.localdomain
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 14.2-RELEASE-p4 stable/25.1-n269832-6addeda7db20 SMP
  Panic String: page fault
  Dump Parity: 3817318220
  Bounds: 2
  Dump Status: good
Here is the /var/crash/info.* from the Backup Firewall:

Code Select Expand

Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 155648
  Blocksize: 512
  Compression: none
  Dumptime: 2025-08-12 22:40:23 +0200
  Hostname: OPNsense2.localdomain
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 14.2-RELEASE-p4 stable/25.1-n269832-6addeda7db20 SMP
  Panic String: page fault
  Dump Parity: 3234046287
  Bounds: 0
  Dump Status: good

Code Select Expand

Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 155648
  Blocksize: 512
  Compression: none
  Dumptime: 2025-08-14 15:37:13 +0200
  Hostname: OPNsense2.localdomain
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 14.2-RELEASE-p4 stable/25.1-n269832-6addeda7db20 SMP
  Panic String: page fault
  Dump Parity: 4270040911
  Bounds: 1
  Dump Status: good

Hi,

We have solved our problem.
https://forum.opnsense.org/index.php?topic=27124.msg131583#msg131583 has provided us with the solution.

We switched from a firewall setup to a high availability setup earlier this year and imported some settings like the users. I know this is not recommended.
After this import, the value of <nextuid> was 2000. And because we imported our user settings, we already had a user with uid 2000.

[Edit:]

Hello,

We have a problem with importing some users or the behavior.

An LDAP server serves as the basis for our user administration. With the appropriate group, the users intended for VPN access can be imported from the LDAP into the firewall.

For colleagues from the helpdesk, we have a group that can access restricted functions of the firewall.

Since the last 2 imported users, however, we have noticed a strange phenomenon. These users were automatically in the helpdesk group.
If we remove user X from the helpdesk group, user A, who is in the group as intended, is also automatically removed.
It looks as if the newly imported user X is linked to user A from the helpdesk group.
If we add user A to the Helpdesk group, user X is automatically in the group again.

This has already been the case with two different users.

What could be the reason for this? Where can I look to find out how these users are linked to each other?
Other users have not yet been affected by this. It only affects new users and those who are assigned to the Helpdesk group.

I hope this is the right section of the forum for this question.


Edit: After some research we found out that the uid starts again at 2002 instead of 2249 where it should be.
For this reason, there are two users with the same uid.

How can this happen and how can we fix it?

The users A and X have different uid's in our LDAP.

Added two screenshots.
Changed the topic title to a more fitting title.

Hi,

It looks like I've solved the problem.

Yesterday I tried to set up High Availability on another hardware machine to rule out a hardware problem. After booting OPNsense from a live stick, I was able to set up High Availability and it worked.

The difference: The live stick is running version 23.1.
The other system was running 23.1.3.1.

So I reinstalled OPNsense and set up High Availability. But again the same problem. But possible on the system with a live stick. So the problem must be with the hard disk?
So I completely wiped the hard disk with an external tool and reinstalled OPNsense. Again the same problem...
But this time I had a working config from the live stick. So I imported it, restarted the system and it works?

Why? What am I missing?


At least I can now continue to test the functions and operation of High Availability.

Thanks to those who tried to help and the input!

Hi anomaly0617,

i have double and triple checked the firewall rules. I allow all IPv4 CARP traffic on all interfaces. On the pfSync Interface I have the rule that allows all traffic.

I have these rules on both firewalls but I still have the same problem :(

Hi mimugmail,

i have removed the lagg and now use a single direct cable connection between the two firewalls but still have the same problem  :(

There is nothing more. There is nothing in the informational log.

The result of the "dmesg -a" is in the attached screenshot.

Hi mimugmail,

i have attached 3 files. Two screenshots of the sync state section and one of the log when applying.

Or do you want a different log? If yes, please specify what log you wanna see.

Thank you for your help!

Hi Patrick,

thank you for your answer.

The HA link is on a dedicated interface with a cable directly between the two firewalls. I am using the address of the other firewall for pfsync.

Unfortunately I'm still learning how to use and interpret tcpdump, but maybe you can help?
I have put the packet capture from the master firewall in the attachments from the moment when I enable "Synchronize States".

Hello,

We would like to use OPNsense with High Availability, but keep running into the following problem during setup.

We are using two identical hardware systems with OPNsense version 24.1.3_1.

The following sources were used as instructions:
- https://docs.opnsense.org/manual/how-tos/carp.html
- https://www.thomas-krenn.com/en/wiki/OPNsense_HA_Cluster_configuration (it's a German website)
- https://www.youtube.com/watch?v=I5n3QXOlxmw

Up to the step "Setup pfSync and HA sync (xmlrpc)" everything works without any problems.

The firewalls communicate with each other.
I can send a ping to 1.1.1.1 and get a response.
I can switch off one firewall and the other firewall takes over immediately.
Everything works as it should.

However, as soon as I check the "Synchronize States" checkbox under "System > High Availability > Settings", it no longer works.
Under "System > High Availability > Status" I get the message "The backup firewall is not accessible or not configured" after waiting a while.
The ping to 1.1.1.1 is lost if the master firewall is not available.

As soon as I remove the tick from the "Synchronize States" checkbox, it works again without any problems.
Firewall 2 takes over if Firewall 1 is not available and vice versa.


I have configured the corresponding interfaces on both firewalls.
I have created the rules for both the sync interface with "Allow all" on both firewalls, as well as a rule for the CARP protocol on the WAN and LAN interface.
I have created the corresponding VIPs on both firewalls.
I have created NAT on both firewalls.


Which settings am I overlooking?

Thank you for any help! If any further information is needed, I will try to provide it.