Messages

In the meantime I have found a workaround that is sufficient for me.

On my WAN firewall I have configured two port forwarding.
Port x is forwarded to the OpenVPN port of Carp Node 1 and port Y is forwarded to the OpenVPN port of Carp Node 2.

So I have two corresponding configurations in my VPN client.

The Topic can be closed from my side.

[I struggle with my current OpenVPN configuration.]

Hi Community.   
   
I run the current OPNsense Version (Version 19.1.6) in a two node Carp Cluster behind my ISP Router.   
DynDNS is configured in my ISP Router.   
Port forwarding ex. Port 80/443 to a host in my network behind the Carp Cluster is working very well by forwarding this traffic to the Carp Cluster virtual IPv4 address.   
   
I struggle with my current OpenVPN configuration.   
I've forwarded the UDP Port 1194 to the Carp Cluster virtual IPv4 address in my ISP Router.   
   
The OpenVPN configuration is similar to the one in the https://wiki.opnsense.org/manual/how-tos/sslvpn_client.html tutorial. The difference is that I use only SSL/TSL + User Auth. Also my transfer and local network is different.   
   
I use Viscosity (1.7.14) on Mac OS (Version 10.14.4) as OpenVPN Client.   
I've done a Client Export from the master OPNsense Node and imported this to my VPN Client.   
   
If I connect a Mac directly to the ISP Router (with DHCP IPv4 from the ISP Router) I can connect to the OpenVPN Server.   
If I try to connect from outside (Internet) the connection always fail.   
   
Verbosity Level of the OpenVPN Server is 3.   

Failed attempts look like:

From the Server log I got this:
01.04.14 01:52   openvpn[3317]: MANAGEMENT: Client disconnected
01.04.14 01:52   openvpn[3317]: MANAGEMENT: CMD 'quit'
01.04.14 01:52   openvpn[3317]: MANAGEMENT: CMD 'status 2'
01.04.14 01:52   openvpn[3317]: MANAGEMENT: Client connected from /var/etc/openvpn/server1.sock
01.04.14 01:51   openvpn[3317]: MANAGEMENT: Client disconnected
01.04.14 01:51   openvpn[3317]: MANAGEMENT: CMD 'quit'
01.04.14 01:51   openvpn[3317]: MANAGEMENT: CMD 'status 2'
01.04.14 01:51   openvpn[3317]: MANAGEMENT: Client connected from /var/etc/openvpn/server1.sock
01.04.14 01:50   openvpn[3317]: MANAGEMENT: Client disconnected
01.04.14 01:50   openvpn[3317]: MANAGEMENT: CMD 'quit'
01.04.14 01:50   openvpn[3317]: MANAGEMENT: CMD 'status 2'
01.04.14 01:50   openvpn[3317]: MANAGEMENT: Client connected from /var/etc/openvpn/server1.sock
01.04.14 01:50   openvpn[3317]: MANAGEMENT: Client disconnected
01.04.14 01:50   openvpn[3317]: MANAGEMENT: CMD 'status 3'
01.04.14 01:50   openvpn[3317]: MANAGEMENT: Client connected from /var/etc/openvpn/server1.sock
   
From the Client log I got this:   
2019-04-14 01:49:42: Viscosity Mac 1.7.14 (1480)
2019-04-14 01:49:42: Viscosity OpenVPN Engine Started
2019-04-14 01:49:42: Running on macOS 10.14.4
2019-04-14 01:49:42: ---------
2019-04-14 01:49:42: State changed to verbinde
2019-04-14 01:49:42: Checking reachability status of connection...
2019-04-14 01:49:42: Connection is reachable. Starting connection attempt.
2019-04-14 01:49:42: OpenVPN 2.4.6 x86_64-apple-darwin [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [MH/RECVDA] [AEAD] built on Nov 23 2018
2019-04-14 01:49:42: library versions: OpenSSL 1.0.2q  20 Nov 2018, LZO 2.10
2019-04-14 01:49:43: TCP/UDP: Preserving recently used remote address: [AF_INET]XXX.XXX.XXX.XXX:1194
2019-04-14 01:49:43: UDP link local (bound): [AF_INET][undef]:0
2019-04-14 01:49:43: UDP link remote: [AF_INET]XXX.XXX.XXX.XXX:1194
2019-04-14 01:50:44: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
2019-04-14 01:50:44: TLS Error: TLS handshake failed
2019-04-14 01:50:44: SIGTERM[soft,tls-error] received, process exiting
2019-04-14 01:50:44: State changed to getrennt
2019-04-14 01:50:45: Viscosity Mac 1.7.14 (1480)
2019-04-14 01:50:45: Viscosity OpenVPN Engine Started
2019-04-14 01:50:45: Running on macOS 10.14.4
2019-04-14 01:50:45: ---------
2019-04-14 01:50:45: State changed to verbinde
2019-04-14 01:50:45: Checking reachability status of connection...
2019-04-14 01:50:45: Connection is reachable. Starting connection attempt.
2019-04-14 01:50:45: OpenVPN 2.4.6 x86_64-apple-darwin [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [MH/RECVDA] [AEAD] built on Nov 23 2018
2019-04-14 01:50:45: library versions: OpenSSL 1.0.2q  20 Nov 2018, LZO 2.10
2019-04-14 01:50:46: TCP/UDP: Preserving recently used remote address: [AF_INET]XXX.XXX.XXX.XXX:1194
2019-04-14 01:50:46: UDP link local (bound): [AF_INET][undef]:0
2019-04-14 01:50:46: UDP link remote: [AF_INET]XXX.XXX.XXX.XXX:1194
2019-04-14 01:51:08: State changed to Disconnecting
2019-04-14 01:51:08: SIGTERM[hard,] received, process exiting
2019-04-14 01:51:08: State changed to getrennt
   
It would be great if someone can support me at this point because I've no idea how to proceed now.   
   
Best regards,   
Rainer

Update:
I spend this evening some time in troubleshooting and I found out that I can only access the Master WAN IP
when I connect from the perimeter network with the OpenVPN client. This means that I cannot connect to the carp cluster virtual IP. Now I've done the port forwarding on my ISP Router to the Master IP and now I can connect from the internet to the OpenVPN Server. So I have to spend some more time to find out if a connection to the carp cluster virtual ip is possible or not.

Hi all.

I want to drop a short note how I found the root cause and how I fixed the problem.

First my system description:

• OPNsense 18.7.10-amd64
• 2-node HA Cluster
• HW: Zotac Zbox Nano

The Problem:
After a power failure of the backup firewall the unbound service didn't start anymore.

When I try to start the service I got the following entry in the 'General' log file:
opnsense: /usr/local/etc/rc.reload_all: The command '/usr/local/sbin/unbound -c '/var/unbound/unbound.conf'' returned exit code '1', the output was '[1547833339] unbound[85235:0] error: Error for server-cert-file: /var/unbound/unbound_server.pem [1547833339] unbound[85235:0] error: Error in SSL_CTX use_certificate_chain_file crypto error:02001002:system library:fopen:No such file or directory [1547833339] unbound[85235:0] error: and additionally crypto error:20074002:BIO routines:FILE_CTRL:system lib [1547833339] unbound[85235:0] error: and additionally crypto error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system lib [1547833339] unbound[85235:0] fatal error: could not set up remote-control'

After opening an ssh session on both host (master and backup firewall) I compared the content of the /var/unbound/ folder and I found out that the following files were missing on the backup firewall:

• unbound_control.key
• unbound_control.pem
• unbound_server.key
• unbound_server.pem

So I tried to run the following command to solve the issue: sudo -u unbound unbound-control-setup -d /var/unbound/

As result I got the following error:
setup in directory /var/unbound/
generating unbound_server.key
/usr/bin/openssl: Undefined symbol "[SL_set_jio"
/usr/local/sbin/unbound-control-setup fatal error: could not genrsa

In the 'General' log file I got this:
opnsense: /usr/local/etc/rc.reload_all: The command 'chroot -u unbound -g unbound / '/usr/local/sbin/unbound-control-setup' -d '/var/unbound'' returned exit code '1', the output was 'setup in directory /var/unbound generating unbound_server.key /usr/bin/openssl: Undefined symbol "[SL_set_jio" /usr/local/sbin/unbound-control-setup fatal error: could not genrsa'


I tried to get the openssl version (/usr/bin/openssl) by using the following command on the backup firewall: openssl version

I got the following error message:
/usr/bin/openssl: Undefined symbol "[SL_set_jio"

At this point I found the root cause because the openssl file was corrupted.
So as workaround I've copied the openssl file from the master to the backup firewall by using scp.

After this I was able to run the command sudo -u unbound unbound-control-setup -d /var/unbound/ successfully and also to start the unbound service on the backup firewall again.

On long term I will re-install the backup firewall because I don't know if other files are corrupted too.

Best regards,
Rainer.

[First of all, here are the key facts of my project:]

Hello Community.

First of all, here are the key facts of my project:

• LAB installation for testing purposes
• 2 OPNsense 17.7.5 Hardware Boxes in a Carp Cluster
• Configured services in OPNsense are:


• DHCP
• Unbound DNS
• Web Proxy
• Network segmentation into 8 subnetworks via VLAN
• An interface is configured for each VLAN (local and VIP)
• Access to the subnets is controlled by appropriate firewall rules
• IPsec Road Warrior VPN is configured (VIP WAN interface)
• Client type is the Cisco IPSec VPN integrated in Mac OS High Sierra
• Via IPsec I can easily access a subnet
• the public IPv4 is provided via DynDNS
• The OPNsense Carp Cluster is behind a router that forwards ports 500 and 4500 UDP to the VIP of the Carp Cluster.
  

After I have enticed the reader of this post to continue reading, here is the essential information about what I want to configure.  ;)

Basically, my configuration works without any problems and has been in operation for about a year.

What exactly does this mean and what does it have to do with the subject of this post?

Right, now it's getting exciting. Currently I can access a subnet via IPsec VPN without any problems.
However, I would like to extend the access to several subnets.

That's the point where I can't move on.
I searched the forum, read the documentation, found some hints, but couldn't find a solution.

Now I have landed in the FUBA (fiddling and tinkering) mode and tested various settings of the IPsec tunnels and the mobile client configuration - so far without success;
Lastly, I had the idea to configure a separate phase-2 entry for each subnet, but that didn't work either.

It would be damn cool if any of you had a solution to my problem.

It would also be cool if someone could tell me if what I'm planning to do is technically possible or not.

Every hint is more than welcome to leave FUBA mode.

If you don't want to answer in english, I am a native german speaker and you can also answer me in german.
Depending on how this post develops, I can write a summary in English and/or German so that other searchers can also benefit from the result.

Best regards,
Rainer.

From my point of view this topic can be closed.

Best regards,
Rainer.

Thank you for your quick reply.

Attached you'll find the configuration differences that were be shown by using the diff function in the history.

Hello Community.

I use my OPNsense boxes - 2 in a Carp Cluster - also as CA in my laboratory environment.

Current version:

• OPNsense 17.7.3-amd64
• FreeBSD 11.0-RELEASE-p12
• OpenSSL 1.0.2l 25 May 2017

Before and after configuration changes to my OPNsense boxes I always archive the configuration.

Yesterday I noticed that except for 4 server certificates, all the others disappeared.
This happened in version 17.7.2, because I just recently upgraded to the current version.
At first I thought that I accidentally - which is actually not possible - deleted the certificates myself.

But I have just checked the saved configurations for both nodes and found out that the certificates were still in the backup file from 13.09.2017.

Unfortunately I have absolutely no idea what could have happened and therefore I cannot reproduce it.

Basically, this is not a problem, because I run another CA in the lab environment, which I can use for all server certificates if necessary.

It would be interesting if I could find out the cause.

Are there any logfiles that I can use for root cause analysis?

Best regards,
Rainer.

Hi ITKOA,

hast du zwischenzeitlich eine Lösung gefunden?

Falls nicht dann beschreib doch mal die Netzwerkkonfiguration von deinen ESXi und füge eine Topologie Skizze hinzu.
Ich würde mir das Szenario dann mal in einer Laborumgebung nachbauen und versuchen dies zu reproduzieren.

Interessant ist auch die VM Konfiguration.
Hast du die VMware Tools installiert, wieviel Netzwerk Adapter hast du konfiguriert, welchen Netzwerk Adaptertyp nutzt du in der VM, welches Gastbetriebssystem hast du für die VMs gewählt, wieviel RAM hast du zugewiesen, wieviel CPU/Cores hast du zugewiesen, ... ?

Grüße,
Rainer.

Hi.

I know this is a weird question but I want to change the startup/shutdown sound of my OPNsense boxes.
The reason is that I like the X-Files sound and want to use this instead of the build in sound.

Looking forward to hearing from you.

Rainer.

Hi.

I've updated both nodes of my HA cluster to from version 17.1.6 to version 17.1.7.

After checking the Dashboard I saw that my Gateway status was unknown on both nodes.
I checked the apinger service and it was down on both nodes and I was unable to start the service.
In the service log there was the messages "apinger: No usable targets found, exiting".

So first I've done a reboot of both nodes but nothing changed.
I was unable to start the apinger service.

At this point I had a weird idea.
I went to System-Gateways-All and select the "edit gateway" action.
I don't changed a setting but I've saved by using the save button.
After applying the setting (strange because I didn't changed anything) the pinger service starts and my Gateway status in the Dashboard changed from unknown to online. :-)

I post this here for people who run into the same issue.

Best regards,
Rainer.

Hi Franco,

I struggled a little bit at the beginning because as a have done the setup of the carp cluster I found out that my VLAN interfaces, the VLANs itself and all VLAN firewall rules from the master didn't appear on the backup node.

At this point I disabled the carp cluster and I've make a backup of my master configuration and restored this to the backup node. Then I changed the backup node name, IPs and so on. During this my backup node was only local connected to a notebook. I've done this because I don't want to make all the configuration again by hand.

After thinking of the further steps I created this topic.
In parallel I designed a configuration sheet for my VLAN configuration.

For my VLAN interfaces I configured on each node a dedicated IP address.
Then I created the corresponding virtual IPs in the Firewall-VirtualIs-Section.
For all IPs I created also a corresponding DNS record on the master box.

The next step was to connect the backup node again to my network.
Then I configured the Carp cluster again and done a reboot of each node.

After this my cluster was running fine without any problem.
I have now the exact VLAN configuration on each node.

I've done also some successfully failover tests by disconnecting the master/backup node from the network.

Best regards,
Rainer.

The Carp Cluster with my VLANs is up and running. :-)

The topic can be closed.

Hi all,

I've extended my OPNsense Zotac Nano CI323 box with an additional one and want to setup a Carp Cluster.

The basic configuration of a Carp Cluster is clear because I found the information in the how to's.
On my master box I've configured 7 VLANs and corresponding interfaces assigned to the VLANs.
As far as I understood the Carp Cluster I have to configure for each VLAN one virtual IP.

Am I right with this because otherwise I will have duplicate IPs on my VLANs?

Looking forward to get some feedback.

Rainer.