Messages

1

High availability / HA Failover Error - Ratelimiting?

« on: September 13, 2024, 09:48:14 am »

Hello,
we've an problem with our HA Setup and stuck there.
Its a 2 node Setup with 24.10.
We enabled state sync.

If we enable maintenance mode on the master the 2nd node takes over the work.
If we try to ping an host while this happens, it works realy well. We got no or just one ping packet that does not reach its destination. So basicly it should be fine.


Problem
But our problem is that after an master switch pretty much all other connections expect icmp arent working well.
We cant see a pattern there independent of port, protocol, ip, vlan or interface connections cant be successfull established.
The strange thing about it is after a while (~30mins) the whole thing settles down and the connections can be successfully established again. We can see in our monitoring system how more and more connections are made the longer we wait.
We can reproduce this every time we switch the master.
It seems to us as if there is some kind of rate limit or something similar at play here and blocks.
The server doesnt got a high load or anything. On the switch side we see nothing special happen. The interfaces arent much utilized too. The states table is also not full.

We've found options in the advanced settings for every firewall rule:

• Max new connections
• Max source states
• Max established
• Max states
• Max source nodes

But these are all untouched / empty for all our rules.

Any Ideas what happens with our setup while Failover and how we can fix it? How can we further analyse it?
In its current state it is unfortunately completely unusable.

2

23.1 Legacy Series / Re: 23.1 PHP recurring exceptions

« on: May 12, 2023, 09:05:09 am »

Thanks for Reply!
Unfortunately we can't get any further with this.
No Error shown.

How can we go ahead?

3

23.1 Legacy Series / 23.1 PHP recurring exceptions

« on: May 10, 2023, 05:44:52 pm »

Hey,
after upgrade to 23.1.6 we got recurring exception reports on one of our nodes.
We automatically copy many of the config settings from one node to another via XMLRPC Sync.
How can we further analyse the root cause of this XML Error?
This behave started soon after upgrading to the new major version. Before we didn't get this problem.
Can we increase loglevel or anything to go ahead?



Additonal notes below:

Code: [Select]

System Information:
User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/112.0
FreeBSD 13.1-RELEASE-p7 stable/23.1-n250430-7eb6eb035df SMP amd64
OPNsense 23.1.6 982c0e24c
Plugins os-node_exporter-1.1 os-wireguard-1.13_5
Time Wed, 10 May 2023 17:14:50 +0200
OpenSSL 1.1.1t  7 Feb 2023
Python 3.9.16
PHP 8.1.18

Code: [Select]

PHP Errors:

[10-May-2023 15:20:00 Europe/Berlin] PHP Fatal error:  Uncaught OPNsense\Core\ConfigException: invalid config xml in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php:383
Stack trace:
#0 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(412): OPNsense\Core\Config->loadFromStream(Resource id #19)
#1 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(725): OPNsense\Core\Config->load()
#2 /usr/local/etc/inc/xmlrpc/legacy.inc(145): OPNsense\Core\Config->lock()
#3 /usr/local/opnsense/contrib/IXR/IXR_Library.php(446): restore_config_section_xmlrpc(Array)
#4 /usr/local/opnsense/contrib/IXR/IXR_Library.php(384): IXR_Server->call('opnsense.restor...', Array)
#5 /usr/local/opnsense/contrib/IXR/IXR_Library.php(357): IXR_Server->serve('__construct(Array)
#7 /usr/local/www/xmlrpc.php(104): XMLRPCServer->start()
#8 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php on line 383
[10-May-2023 15:30:00 Europe/Berlin] PHP Fatal error:  Uncaught OPNsense\Core\ConfigException: invalid config xml in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php:383
Stack trace:
#0 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(412): OPNsense\Core\Config->loadFromStream(Resource id #19)
#1 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(725): OPNsense\Core\Config->load()
#2 /usr/local/etc/inc/xmlrpc/legacy.inc(145): OPNsense\Core\Config->lock()
#3 /usr/local/opnsense/contrib/IXR/IXR_Library.php(446): restore_config_section_xmlrpc(Array)
#4 /usr/local/opnsense/contrib/IXR/IXR_Library.php(384): IXR_Server->call('opnsense.restor...', Array)
#5 /usr/local/opnsense/contrib/IXR/IXR_Library.php(357): IXR_Server->serve('__construct(Array)
#7 /usr/local/www/xmlrpc.php(104): XMLRPCServer->start()
#8 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php on line 383
[10-May-2023 16:20:00 Europe/Berlin] PHP Fatal error:  Uncaught OPNsense\Core\ConfigException: invalid config xml in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php:383
Stack trace:
#0 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(412): OPNsense\Core\Config->loadFromStream(Resource id #19)
#1 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(725): OPNsense\Core\Config->load()
#2 /usr/local/etc/inc/xmlrpc/legacy.inc(145): OPNsense\Core\Config->lock()
#3 /usr/local/opnsense/contrib/IXR/IXR_Library.php(446): restore_config_section_xmlrpc(Array)
#4 /usr/local/opnsense/contrib/IXR/IXR_Library.php(384): IXR_Server->call('opnsense.restor...', Array)
#5 /usr/local/opnsense/contrib/IXR/IXR_Library.php(357): IXR_Server->serve('__construct(Array)
#7 /usr/local/www/xmlrpc.php(104): XMLRPCServer->start()
#8 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php on line 383
[10-May-2023 16:50:00 Europe/Berlin] PHP Fatal error:  Uncaught OPNsense\Core\ConfigException: invalid config xml in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php:383
Stack trace:
#0 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(412): OPNsense\Core\Config->loadFromStream(Resource id #19)
#1 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(725): OPNsense\Core\Config->load()
#2 /usr/local/etc/inc/xmlrpc/legacy.inc(145): OPNsense\Core\Config->lock()
#3 /usr/local/opnsense/contrib/IXR/IXR_Library.php(446): restore_config_section_xmlrpc(Array)
#4 /usr/local/opnsense/contrib/IXR/IXR_Library.php(384): IXR_Server->call('opnsense.restor...', Array)
#5 /usr/local/opnsense/contrib/IXR/IXR_Library.php(357): IXR_Server->serve('__construct(Array)
#7 /usr/local/www/xmlrpc.php(104): XMLRPCServer->start()
#8 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php on line 383

Thanks for your help!

4

Virtual private networks / Wireguard performance many Users

« on: May 03, 2023, 04:00:49 pm »

Hello,
are there any experiences with an WG-Setup and many WG Users (2000+)?
How good is the perfomance?
Are the implementation designed for things like this?
Any limitations?
Does we need to create multiple WG-Interfaces? Does that make any difference?

We are consider our current options.
Our Hardware looks like: AMD EPYC 7272, 32 GB Ram.

5

22.7 Legacy Series / Re: IDS Alert View

« on: February 08, 2023, 10:23:25 am »

No Traffic isnt from the FW itself.

well, the more technically correct answer would be: a packet arrives at an interface with a name that cannot be resolved into a friendly interface name. but maybe it can't be converted because the code was not updated when the suricata template was updated a few years ago )
I'll try to make a pr. thanks for noticing

So that is the return channel an thats why source/dest column are switched?

yes. rules specify the direction of traffic

Okay, thanks for your confirmation.

6

22.7 Legacy Series / Re: IDS Alert View

« on: February 08, 2023, 09:12:04 am »

Why is the Interface row empty on 50% of the alerts?

traffic originated by firewall host? (like dns queries)

Tanks for your answer!
Other examples below. No Traffic isnt from the FW itself. That internal IP are an other Host.

Code: [Select]

2023-02-08T08:37:47.488728+0100 2000428 blocked 209.197.3.8 80 192.168.28.2 54650 ET POLICY ZIP file download
2023-02-08T08:37:43.633795+0100 2000428 blocked 178.79.242.0 80 192.168.28.2 54648 ET POLICY ZIP file download

traffic from server (2.16.186.27   80) to client (192.168.17.21   50532) contains Server header with missing nginx version.

Didnt get it. Other example:

Code: [Select]

2023-02-08T08:24:23.581517+0100 2028371 blocked 192.168.28.2 54610 52.48.126.58 443 ET JA3 Hash - Possible Malware - Fake Firefox Font Update
2023-02-08T08:24:23.581517+0100 2028371 blocked 192.168.28.2 54610 52.48.126.58 443 ET JA3 Hash - Possible Malware - Fake Firefox Font Update
2023-02-08T08:24:16.373773+0100 2000428 blocked 178.79.242.128 80 192.168.28.2 54608 ET POLICY ZIP file download
2023-02-08T08:24:16.373773+0100 2000428 blocked 178.79.242.128 80 192.168.28.2 54608 ET POLICY ZIP file download
The internal IP isnt reachable from public.
To clarify:
The first 2 lines show the request from the internal Server via port 54610 to external Server on port 80.
The last 2 lines show an awnser from external server via port 80 to the internal vm on port 54608. So that is the return channel an thats why source/dest column are switched?

7

22.7 Legacy Series / IDS Alert View

« on: February 07, 2023, 09:48:54 am »

Hey Guys,
we got an understanding problem in the alert view.
We enabled IDS on all our Interfaces. And set all internal Networks as Home networks.

In the Alertview are for example entrys like:

Code: [Select]

2023-02-07T09:34:25.605081+0100 2008064 blocked 2.16.186.27 80 192.168.17.21 50532 ET DELETED Nginx Server with no version string - Often Hostile Traffi
or

Code: [Select]

2023-02-07T09:33:21.699912+0100 2027390 blocked 192.168.28.2 49791 104.108.184.217 80 ET USER_AGENTS Microsoft Device Metadata Retrieval Client User-Agent
How are these entrys to interpret?

For the first line. The IP "2.16.186.27" is the source. How can it be reach our internal "192.168.17.21" IP Address on port 80.
That can never happen. This IP is internal and cant be accessed from external. What should us these line say?

For the secound line. The IP "192.168.28.2" trying to reach "104.108.184.217" on Port 80. That make sense for us.


Further Question:
Why is the Interface row empty on 50% of the alerts?

8

Intrusion Detection and Prevention / Re: IDS enabled HighPing / Timeout

« on: February 02, 2023, 09:49:00 am »

We have the same problem on one of our installations.
can you specify what you did to your dns config?

We installed the Server in an complete fresh environment which didnt contains an own DNS-Server at this moment. So we add an public dns server entry on the install dialoge.
We forgot to change that setting, so no internale name cant be resolved from opnsense.

9

Intrusion Detection and Prevention / Re: IDS enabled HighPing / Timeout

« on: January 31, 2023, 04:37:17 pm »

Solved the issue.
The DNS-Server in the Opnsense was wrong. So it cant resolve internal DNS-Records correctly.
After solved this IDS works fine without high CPU-Load or highpings.

10

Intrusion Detection and Prevention / IDS enabled HighPing / Timeout

« on: January 10, 2023, 11:14:10 am »

Hey,
we are using OPNsense 22.7.10_2-amd64 on an AMD EPYC 7272 12-Core Processor (12 cores, 24 threads).
We enabled IDS with disabled Promiscuous Mode and Logging. As pattern matcher we setup hyperscan. Hardware offload is disabled.
IDS is restricted to the internet / uplink Interface and specific networks. The Speed is gigabit.
We've downloaded and enabled all rules.

We can reproduce that after enabling IDS and waiting for 5-15mins we got highpings and timeouts every few pings between hosts.
Its unusable for us at this state. This Hardware should be play with IDS easily we think?

How can we prevent this? What are we missing?

Thanks for Ideas!

11

High availability / Re: HA with 3 nodes

« on: September 16, 2022, 12:20:57 pm »

I was just about to ask this too!

It seems trivial to do with CARP, but I don't see a way to sync any of the info to more than 1 node using PFSync

For me its not perfect, but it works.
Sync the config from Node 1 to 2 and from 2 to 3 works fine. You just have to be careful with changes while Node1 is offline.
For the connectionsync (pfsync) just use a multicast network.

So there are no node limits.

12

High availability / HA with 3 nodes

« on: September 08, 2022, 11:51:30 am »

Hello,
is it possible to create an 3 node HA stack?

From here it sounds like possible

OPNsense utilizes the Common Address Redundancy Protocol or CARP for hardware failover. Two or more firewalls can be configured as a failover group. If one interface fails on the primary or the primary goes offline entirely, the secondary becomes active.

https://docs.opnsense.org/manual/hacarp.html?highlight=high

But i cant find any example, questions or topics for this.
How can we achieve this? Are there any guide we can follow? Any experiences?

Thanks for Help!