Topics

Hello :) we've recently switched around a bit some of our network architecture and went from one opnsense box behind a modem using pppoe passthrough to a ha-setup behind a router. Said router has port forwarding enabled, since the firewall on it cannot be disabled.

Using the old setup creating certificates worked just fine. 1 domain and a few SANs. Now it always fails, tested with 18.7.1, 18.7 and 18.1.10 - acme.sh 2.7.9 and 2.7.8 (the old setup was running a 17.7.12 with acme.sh 1.13)

I've uploaded a redacted log of our ha-primary, running 18.1.10 with acme.sh 2.7.8, to https://file.io/muHdvl - if somehow would be so kind as to have a look... our ha-secondary is already on 18.7.1 with acme.sh 2.7.9 - I can upload a log from that system as well.
It does show some errors, but I don't know where I might have gone wrong. I even temporarily allowed all traffic to the https port, which, to me it, rules out the firewall as the source of this problem.
I also have checked the A and CNAME records, they are correct and there is no AAAA record.

I'm trying different approaches, but so far failed to get somewthing working put together.
Architecture would be something a HA-Setup connected to a PPPoE router on each site. So there is HA for internet access which is pretty seemless, is it possible to achieve something equally seamless for VPNs?

From what I´ve read and tried I recon that:
1) IPsec would need two tunnels per HA box, so four for site-to-site, and I don´t have an idea on how to make the boxes failover to the other tunnel.
2) OpenVPN needs a central server? Client failover seems to be possible, but what about server failover?
3a) ZeroTier seems promissing, but using CARP sometimes works and then doesn´t. When it was working and I tested the failover it stopped working completely. And there is not a lot of documentation on that to work from.
3b) ZeroTier with OSPF seems to be another possibility, but from what I gathered it would take to long to switch to the other route to call it seemless?

Some may ask why I need seemless, well, we have, among other things, SIP traffic running over those tunnels and the calls may not be disconnected.

[Host R -> FW L: noHost R -> Host L: no]

So far this seems like a pretty straightforward thing to do, but it's only working partitially and I have no clue as to why. And I got lost in way to many posts like this one and helpful sites explaining stuff about ipsec. I really hope someone here can shed some light on this.

setup:
remote site | host R --- FW R === WAN_NAT L == FW L --- host L | local site

Net R: host R, FW R
Net DMZ: WAN_NAT L, FW L
Net L: FW L, host L

FW R:
* single opnsense box
* local ident: FW R public ip (static)
* remote ident: WAN_NAT L public ip (static)

WAN_NAT L:
* business dsl router only used for pppoe
* DMZ: CARP IP NET DMZ FW L
* NAT from NET DMZ to public ip (static)

FW_L:
* HA-setup with tunnel originating from host-ip (not the carp virtual ip)
* NAT from CARP in NET L to CARP in NET DMZ
* Outbound NAT: NET L to ! NET L via CARP IP NET L FW L
* local ident: WAN_NAT L public ip (static)
* remote ident: FW R public ip (static)

fw rules (FW R & FW L):
* allow everything on IF ipsec

connectivity (icmp/ssh/https):
Host R -> FW L: no
Host R -> Host L: no
Host L -> FW R: yes
Host L -> Host R: yes
FW R -> FW L: no
FW L -> FW R: no

FW R "ipsec statusall"

Code Select Expand


Status of IKE charon daemon (weakSwan 5.6.3, FreeBSD 11.1-RELEASE-p10, amd64):
  uptime: 45 minutes, since Jun 28 13:44:15 2018
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 3
  loaded plugins: charon aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf curve25519 xcbc cmac hmac gcm attr kernel-pfkey kernel-pfroute resolve socket-default stroke vici updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls eap-ttls eap-peap xauth-generic whitelist addrblock counters
Listening IP addresses:
  <public IP FW R>
  <NET R IP FW R>
Connections:
        con1:  <public IP FW R>...<public IP FW L>  IKEv2, dpddelay=30s
        con1:   local:  [<public IP FW R>] uses pre-shared key authentication
        con1:   remote: [<public IP FW L>] uses pre-shared key authentication
        con1:   child:  <NET R> === <NET L> TUNNEL, dpdaction=clear
Security Associations (1 up, 0 connecting):
        con1[1]: ESTABLISHED 45 minutes ago, <public IP FW R>[<public IP FW R>]...<public IP FW L>[<public IP FW L>]
        con1[1]: IKEv2 SPIs: af539d1d52e4b970_i db82b64dfdb13183_r*, pre-shared key reauthentication in 6 hours
        con1[1]: IKE proposal: AES_CBC_256/HMAC_SHA2_512_256/PRF_HMAC_SHA2_512/MODP_2048
        con1{3}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: c9b931a6_i ccc8da4d_o
        con1{3}:  AES_CBC_256/HMAC_SHA2_512_256/MODP_2048, 65187 bytes_i (900 pkts, 0s ago), 152716 bytes_o (929 pkts, 0s ago), rekeying in 39 minutes
        con1{3}:   <NET R> === <NET L>


FW L "ipsec statusall"

Code Select Expand


Status of IKE charon daemon (strongSwan 5.6.3, FreeBSD 11.1-RELEASE-p10, amd64):
  uptime: 47 minutes, since Jun 28 13:44:19 2018
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 10
  loaded plugins: charon aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf curve25519 xcbc cmac hmac gcm attr kernel-pfkey kernel-pfroute resolve socket-default stroke vici updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls eap-ttls eap-peap xauth-generic whitelist addrblock counters
Listening IP addresses:
  <NET DMZ IP FW L>
  <NET DMZ CARP IP FW L>
  <HA SYNC IP>
  <NET L IP FW L>
  <CARP IP NET L FW L>
Connections:
        con1:  <NET DMZ IP FW L>...<public IP FW R>  IKEv2, dpddelay=300s
        con1:   local:  [<public IP FW L>] uses pre-shared key authentication
        con1:   remote: [<public IP FW R>] uses pre-shared key authentication
        con1:   child:  <NET L> === <NET R> TUNNEL, dpdaction=clear
Security Associations (1 up, 0 connecting):
        con1[1]: ESTABLISHED 47 minutes ago, <NET DMZ IP FW L>[<public IP FW L>]...<public IP FW R>[<public IP FW R>]
        con1[1]: IKEv2 SPIs: af539d1d52e4b970_i* db82b64dfdb13183_r, pre-shared key reauthentication in 6 hours
        con1[1]: IKE proposal: AES_CBC_256/HMAC_SHA2_512_256/PRF_HMAC_SHA2_512/MODP_2048
        con1{5}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: ccc8da4d_i c9b931a6_o
        con1{5}:  AES_CBC_256/HMAC_SHA2_512_256/MODP_2048, 1509939 bytes_i (2455 pkts, 1s ago), 301572 bytes_o (1971 pkts, 1s ago), rekeying in 37 minutes
        con1{5}:   <NET L> === <NET R>


What seems to be a bit weird to me is that there is no part like

Code Select Expand


Routed Connections:
        con1{2}:  ROUTED, TUNNEL, reqid 2
        con1{2}:   <NET X> === <NET Y>

like on another ipsec connection we have running. Is this part even needed?


So, in the end it comes down to two questions:
1) what went wrong regarding connectivity?
2) (bonus) how do I get a failover tunnel from an HA-FW to another site?

If there is more info needed I would be happy to provide that. All boxes are running 18.1.10 btw.

When I issue/renew a certificate and use HTTP-01 for validation I get a lot of log output, which is helpful.
When I use DNS-01 instead I get no log lines at all, which demotes debugging to pure guesswork and poking around in the dark does not get my anywhere.

1) Is there a way to see the full acme.sh command that is executed by opnsense, so I can try to get more output executing it myself?
2) Is this an issue with opnsense, or should I open a ticket at the acme.sh github repo?

EDIT: I do get log output, it just vanishes as soon as acme.sh is done running (with whatever result)

So, I found a lot of info on that matter, but I still don´t quite get it. I hope someone here is able to help me with that.

The setup is as follows:
1. we have example.com registered with a hosting provider
2. we have a subdomain sub.example.com with a cname record pointing to a subdomain with a free dyndns provider (freedns) at dyn.example.com
3. lets encrypt is set to DNS-01 using said dyndns provider
4. cert is set to CN sub.example.com and SAN *.sub.example.com

Testing this with the staging environment validation fails. The logs are empty. Any pointers/ideas?

We had a loss of connection for a few hours today to one site and afterwards (and after a reboot) monitoring reported a changed /etc/passwd and /var/log/userlog shows useradd and groupadd for acme, _flowd, dhcpd, squid, zabbix, root and all admin users happening after said reboot. I tried to find something regarding this topic but couldn't. Is this intended behaviour?

Hi, I'm pretty stuck right now so I hope someone here can help me.

We have a site-to-site VPN with IPSec and I can ping the site B box from the site A box, but not vice-versa. Site B clients can ping the site A box and everything beyond, site A clients can ping anything beyond the site B box and the box itself.

On site A the routes to site B's subnets point to site A LAN and as far as I understand it doesn't even matter which interface I choose since the tunnel will notice the traffic is for it and forward it to the other site. Same setup on the other box as well.
Now on to IPSec, both boxes have one phase-1 connection and a few phase-2 connections (one phase-2 connection per subnet).

We DID have an additional problem that not all of site B's clients could connect to servers on site A but that somehow vanished after removing the PPPoE-endpoint we had to put in front of site B because of the PPPoE crash in earlier 17.7 releases. If someone could shed some light onto why that might have happened I'd be quite happy as well. I know it might just be en educated guess, but it might help me understand IPSec a bit better.

We had a very strange issue a few day ago. We were connecting a new office with an OPNSense box (I think 17.7 at that point) via Ethernet on WAN port to a Switch and from there it went via fiber to whereever exactly. It is not DSL, but uses PPPoE. The very moment we plugged in the cable the whole box goes unresponsive, nothing at all, no GUI, no ping - lights on though. It comes back only if we hard reboot it with the cable unplugged.
Sadly there is not much I can do regarding debugging, as we're not on site very often. Currently we're using a different box as a PPPoE endpoint and port forward etc. to our OPNSense box. Next time we're there I can try some things if anyone has ideas, as I would very much prefer to loose the intermediate endpoint, but it might take a while.

Hi, we have a dual stack (IPv4 + IPv6) connection, a modem in bridge mode and pppoe0 on the OPNsense box.
WAN is configured as PPPoE for IPv4 and None for IPv6 since we currently do not make use of IPv6 anyway.
From 17.1 to 17.7 DynDNS was removed from core and is available as a plugin, wich we installed, but it does not work for us anymore.

Service SelfHost:

Code Select Expand


Aug 22 09:17:23 opnsense: /services_dyndns_edit.php: Curl error occurred: bind failed with errno 47: Address family not supported by protocol family
Aug 22 09:17:23 opnsense: /services_dyndns_edit.php: Dynamic DNS (xxxxxx.dyndns.eu): Current Service: selfhost
Aug 22 09:17:23 opnsense: /services_dyndns_edit.php: Dynamic DNS (xxxxxx.dyndns.eu): _checkStatus() starting.
Aug 22 09:16:23 opnsense: /services_dyndns_edit.php: SelfHost: DNS update() starting.
Aug 22 09:16:23 opnsense: /services_dyndns_edit.php: Dynamic DNS (xxxxxx.dyndns.eu): _update() starting.
Aug 22 09:16:23 opnsense: /services_dyndns_edit.php: Dynamic DNS (xxxxxx.dyndns.eu): running get_failover_interface for wan. found pppoe0
Aug 22 09:16:23 opnsense: /services_dyndns_edit.php: Dynamic DNS (xxxxxx.dyndns.eu): YYY.YYY.YYY.YYY extracted
Aug 22 09:16:22 opnsense: /services_dyndns_edit.php: Dynamic DNS: updatedns() starting


Service freeDNS:

Code Select Expand


Aug 22 09:28:27 opnsense: /services_dyndns_edit.php: Curl error occurred: Failed to connect to freedns.afraid.org port 443: Operation timed out


It seems to have something to do with IPv6 (see the last, topmost, log message from curl for selfhost) but that's as far as I get

Hi there!

I am currently testing with OPNsense and the nano image. One of the main reasons I choose this image is the two slices setup that I like very much. The OPNsense website does not hold much information regarding the comparison of full vs nano, however the pfsense docs do (see link below). Now my question is: does that still hold true or did it change since the fork?

Regards, Vince

https://doc.pfsense.org/index.php/Full_Install_and_NanoBSD_Comparison