Messages

Further Config
You cannot view this attachment.
You cannot view this attachment.
You cannot view this attachment.
You cannot view this attachment.

[The complete weird thing we noticed]

Hello,
we want to setup an routebased Ipsec Site2Site VPN to Azure from our Opnsense 24.7.7 with the new strongswan config.
Unfortunately we got some major problems. Azure and Opnsense show the connection is estabilshed but we cant send successful traffic between the networks.
For us it seams that the problem is that the AzureGateway tunnel local address (169.254.0.2) isnt reachable from our Opnsense.

The complete weird thing we noticed is. If we disable the VTI. We can successful send traffic over the VPN even if we disable the route and gateway, as long as we keep the Ipsec-Service enabled. If we stop the service, we cant sent traffic through. If we restart the service it works again. In addition we noticed a packet loss of 30%.
How can this be possible?!



Our topology looks like:
Local Side = Opnsense
Local Network: 192.168.18.0/25
Azure Network: 10.100.2.0/24
Tunnel Network: 169.254.0.0/30

Here are screens from our Configuration:
You cannot view this attachment.
You cannot view this attachment.
You cannot view this attachment.
You cannot view this attachment.



Here are output from the opnsense:

Code Select Expand

root@firewall-02:~ # netstat -rn | grep 10.100.
10.100.2.0/24      169.254.0.2        UGS      ipsec1

root@firewall-02:~ # netstat -rn | grep 169.
10.100.2.0/24      169.254.0.2        UGS      ipsec1
169.254.0.1        link#7             UHS         lo0
169.254.0.2        link#32            UH       ipsec1

root@firewall-02:~ # ping 169.254.0.2
PING 169.254.0.2 (169.254.0.2): 56 data bytes
^C
--- 169.254.0.2 ping statistics ---
6 packets transmitted, 0 packets received, 100.0% packet loss



root@firewall-02:~ # ipsec statusall
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
Status of IKE charon daemon (strongSwan 5.9.14, FreeBSD 14.1-RELEASE-p5, amd64):
  uptime: 4 hours, since Feb 13 11:40:49 2025
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 2
  loaded plugins: charon aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs12 pgp dnskey sshkey pem openssl pkcs8 fips-prf curve25519 xcbc cmac hmac kdf gcm drbg curl attr kernel-pfkey kernel-pfroute resolve socket-default stroke vici updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam whitelist addrblock counters
Listening IP addresses:
XXXX
Connections:
36c26b9b-2e4d-4d06-adab-748b1534ed2b:  our public ip...azure public ip  IKEv2
36c26b9b-2e4d-4d06-adab-748b1534ed2b:   local:  uses pre-shared key authentication
36c26b9b-2e4d-4d06-adab-748b1534ed2b:   remote: uses pre-shared key authentication
90a2b6c9-8228-4a19-878f-44ef0772cb85:   child:  192.168.18.0/25 === 10.100.2.0/24 TUNNEL
Security Associations (1 up, 0 connecting):
36c26b9b-2e4d-4d06-adab-748b1534ed2b[3]: ESTABLISHED 49 minutes ago, our public ip[our public ip]...azure public ip[azure public ip]
36c26b9b-2e4d-4d06-adab-748b1534ed2b[3]: IKEv2 SPIs: 1f3a277449d04e35_i* 6ba351ab189fc286_r, rekeying in 2 hours
36c26b9b-2e4d-4d06-adab-748b1534ed2b[3]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
90a2b6c9-8228-4a19-878f-44ef0772cb85{8}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: c35d318e_i 127bdf6e_o
90a2b6c9-8228-4a19-878f-44ef0772cb85{8}:  AES_CBC_256/HMAC_SHA2_256_128/ECP_384, 261156 bytes_i (3109 pkts, 1s ago), 520556 bytes_o (3337 pkts, 1s ago), rekeying in 9 minutes
90a2b6c9-8228-4a19-878f-44ef0772cb85{8}:   192.168.18.0/25 === 10.100.2.0/24


[Problem]

Hello,
we've an problem with our HA Setup and stuck there.
Its a 2 node Setup with 24.10.
We enabled state sync.

If we enable maintenance mode on the master the 2nd node takes over the work.
If we try to ping an host while this happens, it works realy well. We got no or just one ping packet that does not reach its destination. So basicly it should be fine.


Problem
But our problem is that after an master switch pretty much all other connections expect icmp arent working well.
We cant see a pattern there independent of port, protocol, ip, vlan or interface connections cant be successfull established.
The strange thing about it is after a while (~30mins) the whole thing settles down and the connections can be successfully established again. We can see in our monitoring system how more and more connections are made the longer we wait.
We can reproduce this every time we switch the master.
It seems to us as if there is some kind of rate limit or something similar at play here and blocks.
The server doesnt got a high load or anything. On the switch side we see nothing special happen. The interfaces arent much utilized too. The states table is also not full.

We've found options in the advanced settings for every firewall rule:

• Max new connections
• Max source states
• Max established
• Max states
• Max source nodes

But these are all untouched / empty for all our rules.

Any Ideas what happens with our setup while Failover and how we can fix it? How can we further analyse it?
In its current state it is unfortunately completely unusable.

Thanks for Reply!
Unfortunately we can't get any further with this.
No Error shown.

How can we go ahead?

Hey,
after upgrade to 23.1.6 we got recurring exception reports on one of our nodes.
We automatically copy many of the config settings from one node to another via XMLRPC Sync.
How can we further analyse the root cause of this XML Error?
This behave started soon after upgrading to the new major version. Before we didn't get this problem.
Can we increase loglevel or anything to go ahead?



Additonal notes below:

Code Select Expand

System Information:
User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/112.0
FreeBSD 13.1-RELEASE-p7 stable/23.1-n250430-7eb6eb035df SMP amd64
OPNsense 23.1.6 982c0e24c
Plugins os-node_exporter-1.1 os-wireguard-1.13_5
Time Wed, 10 May 2023 17:14:50 +0200
OpenSSL 1.1.1t  7 Feb 2023
Python 3.9.16
PHP 8.1.18

Code Select Expand

PHP Errors:

[10-May-2023 15:20:00 Europe/Berlin] PHP Fatal error:  Uncaught OPNsense\Core\ConfigException: invalid config xml in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php:383
Stack trace:
#0 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(412): OPNsense\Core\Config->loadFromStream(Resource id #19)
#1 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(725): OPNsense\Core\Config->load()
#2 /usr/local/etc/inc/xmlrpc/legacy.inc(145): OPNsense\Core\Config->lock()
#3 /usr/local/opnsense/contrib/IXR/IXR_Library.php(446): restore_config_section_xmlrpc(Array)
#4 /usr/local/opnsense/contrib/IXR/IXR_Library.php(384): IXR_Server->call('opnsense.restor...', Array)
#5 /usr/local/opnsense/contrib/IXR/IXR_Library.php(357): IXR_Server->serve('__construct(Array)
#7 /usr/local/www/xmlrpc.php(104): XMLRPCServer->start()
#8 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php on line 383
[10-May-2023 15:30:00 Europe/Berlin] PHP Fatal error:  Uncaught OPNsense\Core\ConfigException: invalid config xml in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php:383
Stack trace:
#0 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(412): OPNsense\Core\Config->loadFromStream(Resource id #19)
#1 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(725): OPNsense\Core\Config->load()
#2 /usr/local/etc/inc/xmlrpc/legacy.inc(145): OPNsense\Core\Config->lock()
#3 /usr/local/opnsense/contrib/IXR/IXR_Library.php(446): restore_config_section_xmlrpc(Array)
#4 /usr/local/opnsense/contrib/IXR/IXR_Library.php(384): IXR_Server->call('opnsense.restor...', Array)
#5 /usr/local/opnsense/contrib/IXR/IXR_Library.php(357): IXR_Server->serve('__construct(Array)
#7 /usr/local/www/xmlrpc.php(104): XMLRPCServer->start()
#8 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php on line 383
[10-May-2023 16:20:00 Europe/Berlin] PHP Fatal error:  Uncaught OPNsense\Core\ConfigException: invalid config xml in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php:383
Stack trace:
#0 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(412): OPNsense\Core\Config->loadFromStream(Resource id #19)
#1 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(725): OPNsense\Core\Config->load()
#2 /usr/local/etc/inc/xmlrpc/legacy.inc(145): OPNsense\Core\Config->lock()
#3 /usr/local/opnsense/contrib/IXR/IXR_Library.php(446): restore_config_section_xmlrpc(Array)
#4 /usr/local/opnsense/contrib/IXR/IXR_Library.php(384): IXR_Server->call('opnsense.restor...', Array)
#5 /usr/local/opnsense/contrib/IXR/IXR_Library.php(357): IXR_Server->serve('__construct(Array)
#7 /usr/local/www/xmlrpc.php(104): XMLRPCServer->start()
#8 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php on line 383
[10-May-2023 16:50:00 Europe/Berlin] PHP Fatal error:  Uncaught OPNsense\Core\ConfigException: invalid config xml in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php:383
Stack trace:
#0 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(412): OPNsense\Core\Config->loadFromStream(Resource id #19)
#1 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(725): OPNsense\Core\Config->load()
#2 /usr/local/etc/inc/xmlrpc/legacy.inc(145): OPNsense\Core\Config->lock()
#3 /usr/local/opnsense/contrib/IXR/IXR_Library.php(446): restore_config_section_xmlrpc(Array)
#4 /usr/local/opnsense/contrib/IXR/IXR_Library.php(384): IXR_Server->call('opnsense.restor...', Array)
#5 /usr/local/opnsense/contrib/IXR/IXR_Library.php(357): IXR_Server->serve('__construct(Array)
#7 /usr/local/www/xmlrpc.php(104): XMLRPCServer->start()
#8 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php on line 383


Thanks for your help!

Hello,
are there any experiences with an WG-Setup and many WG Users (2000+)?
How good is the perfomance?
Are the implementation designed for things like this?
Any limitations?
Does we need to create multiple WG-Interfaces? Does that make any difference?

We are consider our current options.
Our Hardware looks like: AMD EPYC 7272, 32 GB Ram.

No Traffic isnt from the FW itself.

well, the more technically correct answer would be: a packet arrives at an interface with a name that cannot be resolved into a friendly interface name. but ;) maybe it can't be converted because the code was not updated when the suricata template was updated a few years ago )
I'll try to make a pr. thanks for noticing

So that is the return channel an thats why source/dest column are switched?

yes. rules specify the direction of traffic

Okay, thanks for your confirmation.

Why is the Interface row empty on 50% of the alerts?

traffic originated by firewall host? (like dns queries)

Tanks for your answer!
Other examples below. No Traffic isnt from the FW itself. That internal IP are an other Host.

Code Select Expand

2023-02-08T08:37:47.488728+0100 2000428 blocked 209.197.3.8 80 192.168.28.2 54650 ET POLICY ZIP file download
2023-02-08T08:37:43.633795+0100 2000428 blocked 178.79.242.0 80 192.168.28.2 54648 ET POLICY ZIP file download

traffic from server (2.16.186.27   80) to client (192.168.17.21   50532) contains Server header with missing nginx version.

Didnt get it. Other example:

Code Select Expand

2023-02-08T08:24:23.581517+0100 2028371 blocked 192.168.28.2 54610 52.48.126.58 443 ET JA3 Hash - Possible Malware - Fake Firefox Font Update
2023-02-08T08:24:23.581517+0100 2028371 blocked 192.168.28.2 54610 52.48.126.58 443 ET JA3 Hash - Possible Malware - Fake Firefox Font Update
2023-02-08T08:24:16.373773+0100 2000428 blocked 178.79.242.128 80 192.168.28.2 54608 ET POLICY ZIP file download
2023-02-08T08:24:16.373773+0100 2000428 blocked 178.79.242.128 80 192.168.28.2 54608 ET POLICY ZIP file download

The internal IP isnt reachable from public.
To clarify:
The first 2 lines show the request from the internal Server via port 54610 to external Server on port 80.
The last 2 lines show an awnser from external server via port 80 to the internal vm on port 54608. So that is the return channel an thats why source/dest column are switched?

Hey Guys,
we got an understanding problem in the alert view.
We enabled IDS on all our Interfaces. And set all internal Networks as Home networks.

In the Alertview are for example entrys like:

Code Select Expand

2023-02-07T09:34:25.605081+0100 2008064 blocked 2.16.186.27 80 192.168.17.21 50532 ET DELETED Nginx Server with no version string - Often Hostile Traffi

or

Code Select Expand

2023-02-07T09:33:21.699912+0100 2027390 blocked 192.168.28.2 49791 104.108.184.217 80 ET USER_AGENTS Microsoft Device Metadata Retrieval Client User-Agent

How are these entrys to interpret?

For the first line. The IP "2.16.186.27" is the source. How can it be reach our internal "192.168.17.21" IP Address on port 80.
That can never happen. This IP is internal and cant be accessed from external. What should us these line say?

For the secound line. The IP "192.168.28.2" trying to reach "104.108.184.217" on Port 80. That make sense for us.


Further Question:
Why is the Interface row empty on 50% of the alerts?

We have the same problem on one of our installations.
can you specify what you did to your dns config?

We installed the Server in an complete fresh environment which didnt contains an own DNS-Server at this moment. So we add an public dns server entry on the install dialoge.
We forgot to change that setting, so no internale name cant be resolved from opnsense.

Solved the issue.
The DNS-Server in the Opnsense was wrong. So it cant resolve internal DNS-Records correctly.
After solved this IDS works fine without high CPU-Load or highpings.

Hey,
we are using OPNsense 22.7.10_2-amd64 on an AMD EPYC 7272 12-Core Processor (12 cores, 24 threads).
We enabled IDS with disabled Promiscuous Mode and Logging. As pattern matcher we setup hyperscan. Hardware offload is disabled.
IDS is restricted to the internet / uplink Interface and specific networks. The Speed is gigabit.
We've downloaded and enabled all rules.

We can reproduce that after enabling IDS and waiting for 5-15mins we got highpings and timeouts every few pings between hosts.
Its unusable for us at this state. This Hardware should be play with IDS easily we think?

How can we prevent this? What are we missing?

Thanks for Ideas!

I was just about to ask this too!

It seems trivial to do with CARP, but I don't see a way to sync any of the info to more than 1 node using PFSync :(

For me its not perfect, but it works.
Sync the config from Node 1 to 2 and from 2 to 3 works fine. You just have to be careful with changes while Node1 is offline.
For the connectionsync (pfsync) just use a multicast network.

So there are no node limits.

Hello,
is it possible to create an 3 node HA stack?

From here it sounds like possible

OPNsense utilizes the Common Address Redundancy Protocol or CARP for hardware failover. Two or more firewalls can be configured as a failover group. If one interface fails on the primary or the primary goes offline entirely, the secondary becomes active.

https://docs.opnsense.org/manual/hacarp.html?highlight=high

But i cant find any example, questions or topics for this.
How can we achieve this? Are there any guide we can follow? Any experiences?

Thanks for Help!