Messages

So, further diving into this and still no solution :/

1. Router has a port-forward 80&443 to opnsense
2. opnsense allows access from external to opnsense:80&443 (GUI is OFF for the WAN_IF)
3. opnsense has a port-forward 80&443 to localhost:43580
4. on localhost:43580 is the lighttpd run by the acme-plugin (which is always running, not just when needed, which I find a little weird)

acme.sh still shows "Timeout during connect", "status: 400" BUT when I access that manually I can download the challenge

Has anyone ideas / pointers as to what could be the issue here?

Soo, I just discovered that I was always running into DNS rebinding check, that 501 error page was all lets encrypt could see and thus could not verify. I've since disabled the GUI on the WAN-port.

Also - I found a file that ins apparently responsible for the redirects (acme_anchor_rules), it redirects port 80 to 40k-something. When I used the packet capture I always see TLS requests from lets-encrypt. The acme.sh.log does list port 80 though - which times out.

Is there a setting that always redirects 80 to 443, or something like that?

Could this possibly be a NAT-issue? NAT is

Code Select Expand

nat on WAN_IF inet from $LOCAL to !LOCAL -> WAN_CARP_IP port 1024:65535

interesting idea, however we have a host (openbsd, acme.sh as well) at another location which is behind a firewall as well, so I guess we can rule out that the system acme.sh runs on needs to have a public IP, I just needs to know what the it's public IP is and needs to be reachable for verification.

[edit:]

Explicit errors that are to find in the linked log file (see first post):

Code Select Expand

[Thu Aug 23 11:16:38 CEST 2018] original='{
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:acme:error:connection",
    "detail": "Fetching http://sub.example.com/.well-known/acme-challenge/FH6K-FkTi402Yxnz4GgGH2QmgQ04ZZ7KGlbWbJ3_vIg: Timeout during connect (likely firewall problem)",
    "status": 400
  },
  "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/KIcdLYd-AGixisDwtryje-eCEjmXPl59j1A2Wj14Nho/162774506",
  "token": "FH6K-FkTi402Yxnz4GgGH2QmgQ04ZZ7KGlbWbJ3_vIg",
  "keyAuthorization": "FH6K-FkTi402Yxnz4GgGH2QmgQ04ZZ7KGlbWbJ3_vIg.Dw8O-XYchKlLNiCK7AvuJE-v2gfYOVv9uF1tfsKz2to",
  "validationRecord": [
    {
      "url": "http://sub.example.com/.well-known/acme-challenge/FH6K-FkTi402Yxnz4GgGH2QmgQ04ZZ7KGlbWbJ3_vIg",
      "hostname": "sub.example.com",
      "port": "80",
      "addressesResolved": [
        "X.X.X.X"
      ],
      "addressUsed": "X.X.X.X"
    }
  ]
}'
I really do not see how the firewall could be an issue here, but maybe someone here knows more about that.

Code Select Expand

[Thu Aug 23 11:16:39 CEST 2018] original='{
  "type": "urn:acme:error:malformed",
  "detail": "Unable to update challenge :: The challenge is not pending.",
  "status": 400
}'

Code Select Expand

[Thu Aug 23 11:16:46 CEST 2018] Diagnosis versions:
openssl:openssl
OpenSSL 1.0.2k-freebsd  26 Jan 2017
apache:
apache doesn't exists.
nginx:
nginx doesn't exists.
socat:
[...]

I did see a 403 in another earlier log too, sadly I seem to have deleted that one already.
edit: got it recreated with 18.7

Code Select Expand

[Fri Aug 24 10:15:03 CEST 2018] original='{
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:acme:error:unauthorized",
    "detail": "Invalid response from http://sub.example.com/.well-known/acme-challenge/<token>: \"\u003c!doctype html\u003e\n\u003c!--[if IE 8 ]\u003e\u003chtml lang=\"en\" class=\"ie ie8 lte9 lt
e8 no-js\"\u003e\u003c![endif]--\u003e\n\u003c!--[if IE 9 ]\u003e\u003chtml lang=\"en\" class=\"",
    "status": 403
  },
  "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/<challenge>/163133057",
  "token": "<token>",
  "keyAuthorization": "<token>.<key>",
  "validationRecord": [
    {
      "url": "http://sub.example.com/.well-known/acme-challenge/<token>",
      "hostname": "sub.example.com",
      "port": "80",
      "addressesResolved": [
        "X.X.X.X"
      ],
      "addressUsed": "X.X.X.X"
    },
    {
      "url": "https://sub.example.com/.well-known/acme-challenge/<token>",
      "hostname": "sub.example.com",
      "port": "443",
      "addressesResolved": [
        "X.X.X.X"
      ],
      "addressUsed": "X.X.X.X"
    },
    {
      "url": "https://sub.example.com/?url=/.well-known/acme-challenge/<token>",
      "hostname": "sub.example.com",
      "port": "443",
      "addressesResolved": [
        "X.X.X.X"
      ],
      "addressUsed": "X.X.X.X"
    }
  ]
}'


I notice there a quite a few views, yet no reply. Could I have done something better in describing my problem, or ... ?

Hello :) we've recently switched around a bit some of our network architecture and went from one opnsense box behind a modem using pppoe passthrough to a ha-setup behind a router. Said router has port forwarding enabled, since the firewall on it cannot be disabled.

Using the old setup creating certificates worked just fine. 1 domain and a few SANs. Now it always fails, tested with 18.7.1, 18.7 and 18.1.10 - acme.sh 2.7.9 and 2.7.8 (the old setup was running a 17.7.12 with acme.sh 1.13)

I've uploaded a redacted log of our ha-primary, running 18.1.10 with acme.sh 2.7.8, to https://file.io/muHdvl - if somehow would be so kind as to have a look... our ha-secondary is already on 18.7.1 with acme.sh 2.7.9 - I can upload a log from that system as well.
It does show some errors, but I don't know where I might have gone wrong. I even temporarily allowed all traffic to the https port, which, to me it, rules out the firewall as the source of this problem.
I also have checked the A and CNAME records, they are correct and there is no AAAA record.

I'm trying different approaches, but so far failed to get somewthing working put together.
Architecture would be something a HA-Setup connected to a PPPoE router on each site. So there is HA for internet access which is pretty seemless, is it possible to achieve something equally seamless for VPNs?

From what I´ve read and tried I recon that:
1) IPsec would need two tunnels per HA box, so four for site-to-site, and I don´t have an idea on how to make the boxes failover to the other tunnel.
2) OpenVPN needs a central server? Client failover seems to be possible, but what about server failover?
3a) ZeroTier seems promissing, but using CARP sometimes works and then doesn´t. When it was working and I tested the failover it stopped working completely. And there is not a lot of documentation on that to work from.
3b) ZeroTier with OSPF seems to be another possibility, but from what I gathered it would take to long to switch to the other route to call it seemless?

Some may ask why I need seemless, well, we have, among other things, SIP traffic running over those tunnels and the calls may not be disconnected.

Did you check this already? https://docs.opnsense.org/manual/how-tos/ipsec-s2s-binat.html


Cheers,
Franco

No, I did not figure I need to, since I do not have the same network in use in multiple locations. Do I still need to use BINAT, and if so, why?

[Host R -> FW L: noHost R -> Host L: no]

So far this seems like a pretty straightforward thing to do, but it's only working partitially and I have no clue as to why. And I got lost in way to many posts like this one and helpful sites explaining stuff about ipsec. I really hope someone here can shed some light on this.

setup:
remote site | host R --- FW R === WAN_NAT L == FW L --- host L | local site

Net R: host R, FW R
Net DMZ: WAN_NAT L, FW L
Net L: FW L, host L

FW R:
* single opnsense box
* local ident: FW R public ip (static)
* remote ident: WAN_NAT L public ip (static)

WAN_NAT L:
* business dsl router only used for pppoe
* DMZ: CARP IP NET DMZ FW L
* NAT from NET DMZ to public ip (static)

FW_L:
* HA-setup with tunnel originating from host-ip (not the carp virtual ip)
* NAT from CARP in NET L to CARP in NET DMZ
* Outbound NAT: NET L to ! NET L via CARP IP NET L FW L
* local ident: WAN_NAT L public ip (static)
* remote ident: FW R public ip (static)

fw rules (FW R & FW L):
* allow everything on IF ipsec

connectivity (icmp/ssh/https):
Host R -> FW L: no
Host R -> Host L: no
Host L -> FW R: yes
Host L -> Host R: yes
FW R -> FW L: no
FW L -> FW R: no

FW R "ipsec statusall"

Code Select Expand


Status of IKE charon daemon (weakSwan 5.6.3, FreeBSD 11.1-RELEASE-p10, amd64):
  uptime: 45 minutes, since Jun 28 13:44:15 2018
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 3
  loaded plugins: charon aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf curve25519 xcbc cmac hmac gcm attr kernel-pfkey kernel-pfroute resolve socket-default stroke vici updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls eap-ttls eap-peap xauth-generic whitelist addrblock counters
Listening IP addresses:
  <public IP FW R>
  <NET R IP FW R>
Connections:
        con1:  <public IP FW R>...<public IP FW L>  IKEv2, dpddelay=30s
        con1:   local:  [<public IP FW R>] uses pre-shared key authentication
        con1:   remote: [<public IP FW L>] uses pre-shared key authentication
        con1:   child:  <NET R> === <NET L> TUNNEL, dpdaction=clear
Security Associations (1 up, 0 connecting):
        con1[1]: ESTABLISHED 45 minutes ago, <public IP FW R>[<public IP FW R>]...<public IP FW L>[<public IP FW L>]
        con1[1]: IKEv2 SPIs: af539d1d52e4b970_i db82b64dfdb13183_r*, pre-shared key reauthentication in 6 hours
        con1[1]: IKE proposal: AES_CBC_256/HMAC_SHA2_512_256/PRF_HMAC_SHA2_512/MODP_2048
        con1{3}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: c9b931a6_i ccc8da4d_o
        con1{3}:  AES_CBC_256/HMAC_SHA2_512_256/MODP_2048, 65187 bytes_i (900 pkts, 0s ago), 152716 bytes_o (929 pkts, 0s ago), rekeying in 39 minutes
        con1{3}:   <NET R> === <NET L>


FW L "ipsec statusall"

Code Select Expand


Status of IKE charon daemon (strongSwan 5.6.3, FreeBSD 11.1-RELEASE-p10, amd64):
  uptime: 47 minutes, since Jun 28 13:44:19 2018
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 10
  loaded plugins: charon aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf curve25519 xcbc cmac hmac gcm attr kernel-pfkey kernel-pfroute resolve socket-default stroke vici updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls eap-ttls eap-peap xauth-generic whitelist addrblock counters
Listening IP addresses:
  <NET DMZ IP FW L>
  <NET DMZ CARP IP FW L>
  <HA SYNC IP>
  <NET L IP FW L>
  <CARP IP NET L FW L>
Connections:
        con1:  <NET DMZ IP FW L>...<public IP FW R>  IKEv2, dpddelay=300s
        con1:   local:  [<public IP FW L>] uses pre-shared key authentication
        con1:   remote: [<public IP FW R>] uses pre-shared key authentication
        con1:   child:  <NET L> === <NET R> TUNNEL, dpdaction=clear
Security Associations (1 up, 0 connecting):
        con1[1]: ESTABLISHED 47 minutes ago, <NET DMZ IP FW L>[<public IP FW L>]...<public IP FW R>[<public IP FW R>]
        con1[1]: IKEv2 SPIs: af539d1d52e4b970_i* db82b64dfdb13183_r, pre-shared key reauthentication in 6 hours
        con1[1]: IKE proposal: AES_CBC_256/HMAC_SHA2_512_256/PRF_HMAC_SHA2_512/MODP_2048
        con1{5}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: ccc8da4d_i c9b931a6_o
        con1{5}:  AES_CBC_256/HMAC_SHA2_512_256/MODP_2048, 1509939 bytes_i (2455 pkts, 1s ago), 301572 bytes_o (1971 pkts, 1s ago), rekeying in 37 minutes
        con1{5}:   <NET L> === <NET R>


What seems to be a bit weird to me is that there is no part like

Code Select Expand


Routed Connections:
        con1{2}:  ROUTED, TUNNEL, reqid 2
        con1{2}:   <NET X> === <NET Y>

like on another ipsec connection we have running. Is this part even needed?


So, in the end it comes down to two questions:
1) what went wrong regarding connectivity?
2) (bonus) how do I get a failover tunnel from an HA-FW to another site?

If there is more info needed I would be happy to provide that. All boxes are running 18.1.10 btw.

When I issue/renew a certificate and use HTTP-01 for validation I get a lot of log output, which is helpful.
When I use DNS-01 instead I get no log lines at all, which demotes debugging to pure guesswork and poking around in the dark does not get my anywhere.

1) Is there a way to see the full acme.sh command that is executed by opnsense, so I can try to get more output executing it myself?
2) Is this an issue with opnsense, or should I open a ticket at the acme.sh github repo?

EDIT: I do get log output, it just vanishes as soon as acme.sh is done running (with whatever result)

So, I found a lot of info on that matter, but I still don´t quite get it. I hope someone here is able to help me with that.

The setup is as follows:
1. we have example.com registered with a hosting provider
2. we have a subdomain sub.example.com with a cname record pointing to a subdomain with a free dyndns provider (freedns) at dyn.example.com
3. lets encrypt is set to DNS-01 using said dyndns provider
4. cert is set to CN sub.example.com and SAN *.sub.example.com

Testing this with the staging environment validation fails. The logs are empty. Any pointers/ideas?

Well, for me it's because all other VPNs we have are IPsec based and I'd rather stay with one solution. It's not all OPNsense yet either. And in this case it's much easier to just add a new ppoe endpoint instead of switching everything to OpenVPN, although I would have liked to skip having to use (and secure) another device.
In general, as far as I remember, someone might want that because IPsec has more features than OpenVPN, although it sure can be a real pain to work with sometimes.

Is this possible with IPsec as well? I guess in a way it must be, since Sophos seems to provide that, but I have (and want) to stay with our OPNsense boxes. Setup see below. Currently this setup is running without the HA-2 part and switch, with an IPsec VPN inbetween HA-1 and FW-3. I was just wondering if I could make the failover happen without putting another box in that serves as a pppoe endpoint for site 1.

Code Select Expand


                                                               site 1 #                      #  site 2
            |---[ HA-1 ]---pppoe1---|                                 #                      #
{ LAN-1 }---|       |               |---[ switch ]---[ eth to opt ]---#---[ pppoe server ]---#---pppoe3---[ DSL modem ]---[ FW-3 ]---{ LAN-2 }
            |---[ HA-2 ]---pppoe2---|                                 #                      #


Hi franco, thanks for clearing that up! :)

We had a loss of connection for a few hours today to one site and afterwards (and after a reboot) monitoring reported a changed /etc/passwd and /var/log/userlog shows useradd and groupadd for acme, _flowd, dhcpd, squid, zabbix, root and all admin users happening after said reboot. I tried to find something regarding this topic but couldn't. Is this intended behaviour?