26.1.7_2: issue with ACME client automation upload to TrueNAS websocket API

[franco]

In either case it looks like it expects TrueNAS as OS, not OPNsense.


Cheers,
Franco

[Rene78]

[Thu May 14 14:56:38 CEST 2026] Checking TrueNAS health...
/usr/local/sbin/acme.sh: midclt: not found

It seems to be a part of this:
https://github.com/truenas/api_client

Thanks for finding this apparently missing API client. I reproduced this missing midclt also from a shell. Not sure why it does not show up in the system logs though. ;-)

In either case it looks like it expects TrueNAS as OS, not OPNsense.

Cheers,
Franco

What's in a name... the TrueNAS client apparently called midcli and the one ceel references (https://github.com/truenas/api_client) is midclt. To make it more confusing both the clients seem to be preinstalled on a TrueNAS box according to the GitHub documentation.

Anyway,

• the midcli is the NAS cli interface —> that is preinstalled and can only be used (in my understanding) on the TrueNAS box.
• The midclt is not installed by default, at least not on my TrueNAS scale box (25.10.3.1). It is compatible with TrueNAS SCALE (Debian based) according to the docs both also to run from non-TrueNAS clients. "TrueNAS comes with this client preinstalled, but it is also possible to use the TrueNAS websocket client from a non-TrueNAS host."

Hence, imho the midclt could be used (assumption here, if the code works on FreeBSD...) to complete the deployment task.

[Patrick M. Hausen]

Shouldn't the automation "simply" use SSH to execute whatever is necessary on the TrueNAS system, including midclt?

[Rene78]

Shouldn't the automation "simply" use SSH to execute whatever is necessary on the TrueNAS system, including midclt?

I am not skilled enough to check what the acme client and plugin does on OPNSense (use local midctl OR remote calling through SSH or something else HTTP calls etc.). What I understand is that midctl is/was intended to make calling the TrueNAS API as easy as possible.

[Monviech (Cedrik)]

Sadly it's just mid.

(Sorry had to make that joke.)

[TheCrackedCube]

Hi,

  Did anyone find a solution to this?

[amichel]

I "fixed" it by using the ACME plugin on the truenas scale box. I am using easydns as provider and decided to go for a script to set the DNS records. after some fiddling I was able to get that working.

Code Select Expand

#!/bin/sh

# ============================================================
# TrueNAS SCALE ACME Shell Authenticator for easyDNS
#
# Script:
#   /mnt/Pool/Scripts/ACME/acmedns.sh
#
#:
#
# TrueNAS-Format:
#   acmedns.sh set   scale.domain.com _acme-challenge.scale.domain.com TOKEN
#   acmedns.sh unset scale.domain.com _acme-challenge.scale.domain.com TOKEN
#
# Fallback-Format:
#   acmedns.sh set   _acme-challenge.scale.domain.com TOKEN
#   acmedns.sh unset _acme-challenge.scale.domain.com TOKEN
# ============================================================

set -eu

LOG_DIR="/mnt/Pool/Scripts/ACME"
LOG_FILE="${LOG_DIR}/acmedns.log"
MAX_LOG_BYTES=$((1024 * 1024))

API_BASE="https://rest.easydns.net"
EASYDNS_DOMAIN="domain.com"
EASYDNS_TTL=300

EASYDNS_TOKEN_FILE="/mnt/Pool/Scripts/ACME/easydns.token"
EASYDNS_KEY_FILE="/mnt/Pool/Scripts/ACME/easydns.key"

PROPAGATION_SLEEP=60

mkdir -p "$LOG_DIR"

rotate_log_if_needed() {
  if [ -f "$LOG_FILE" ]; then
    size="$(stat -c%s "$LOG_FILE" 2>/dev/null || echo 0)"
    if [ "$size" -gt "$MAX_LOG_BYTES" ]; then
      : > "$LOG_FILE"
    fi
  fi
}

log() {
  echo "[$(date '+%F %T')] $*" >> "$LOG_FILE"
}

fail() {
  log "ERROR: $*"
  exit 1
}

rotate_log_if_needed

[ -f "$EASYDNS_TOKEN_FILE" ] || fail "Missing EasyDNS token file: $EASYDNS_TOKEN_FILE"
[ -f "$EASYDNS_KEY_FILE" ] || fail "Missing EasyDNS key file: $EASYDNS_KEY_FILE"

EASYDNS_TOKEN="$(tr -d '\r\n' < "$EASYDNS_TOKEN_FILE")"
EASYDNS_KEY="$(tr -d '\r\n' < "$EASYDNS_KEY_FILE")"

[ -n "$EASYDNS_TOKEN" ] || fail "EasyDNS token file is empty"
[ -n "$EASYDNS_KEY" ] || fail "EasyDNS key file is empty"

record_to_host() {
  record="$1"

  # trailing dot entfernen und lowercase
  record="$(printf '%s' "$record" | sed 's/\.$//' | tr '[:upper:]' '[:lower:]')"

  case "$record" in
    *."$EASYDNS_DOMAIN")
      printf '%s\n' "${record%.$EASYDNS_DOMAIN}"
      ;;
    "$EASYDNS_DOMAIN")
      printf '%s\n' "@"
      ;;
    *)
      fail "Challenge record '$record' is not inside zone '$EASYDNS_DOMAIN'"
      ;;
  esac
}

json_body_for_add() {
  host="$1"
  token="$2"

  python3 - "$host" "$EASYDNS_DOMAIN" "$EASYDNS_TTL" "$token" <<'PY'
import sys, json

host = sys.argv[1]
domain = sys.argv[2]
ttl = int(sys.argv[3])
token = sys.argv[4]

body = {
    "host": host,
    "domain": domain,
    "ttl": ttl,
    "prio": 0,
    "type": "txt",
    "rdata": token,
}

print(json.dumps(body, separators=(",", ":")))
PY
}

api_get_records() {
  curl -fsS \
    -u "${EASYDNS_TOKEN}:${EASYDNS_KEY}" \
    --connect-timeout 15 \
    --max-time 60 \
    "${API_BASE}/zones/records/all/${EASYDNS_DOMAIN}?format=json"
}

api_add_txt() {
  body="$1"

  curl -fsS \
    -u "${EASYDNS_TOKEN}:${EASYDNS_KEY}" \
    -X PUT \
    --connect-timeout 15 \
    --max-time 60 \
    -H "Content-Type: application/json" \
    -d "$body" \
    "${API_BASE}/zones/records/add/${EASYDNS_DOMAIN}/txt?format=json"
}

api_delete_record() {
  id="$1"

  log "Trying DELETE with confirm body for id=$id"

  if curl -fsS \
    -u "${EASYDNS_TOKEN}:${EASYDNS_KEY}" \
    -X DELETE \
    --connect-timeout 15 \
    --max-time 60 \
    -H "Content-Type: application/json" \
    -d '{"confirm":"DELETE"}' \
    "${API_BASE}/zones/records/${EASYDNS_DOMAIN}/${id}?format=json" >> "$LOG_FILE" 2>&1; then
    log "DELETE with confirm body succeeded for id=$id"
    return 0
  fi

  log "DELETE with confirm body failed for id=$id, trying fallback DELETE without body"

  curl -fsS \
    -u "${EASYDNS_TOKEN}:${EASYDNS_KEY}" \
    -X DELETE \
    --connect-timeout 15 \
    --max-time 60 \
    "${API_BASE}/zones/records/${EASYDNS_DOMAIN}/${id}?format=json" >> "$LOG_FILE" 2>&1

  log "Fallback DELETE succeeded for id=$id"
}

find_txt_record_ids() {
  host="$1"
  token="${2:-}"
  records_json="$3"

  printf '%s' "$records_json" | python3 - "$host" "$token" <<'PY'
import sys, json

host = sys.argv[1].lower()
token = sys.argv[2]

try:
    data = json.load(sys.stdin)
except Exception as e:
    print(f"JSON_ERROR:{e}", file=sys.stderr)
    sys.exit(2)

for r in data.get("data", []):
    rtype = str(r.get("type", "")).lower()
    rhost = str(r.get("host", "")).lower()

    rrdata = r.get("rdata", r.get("rData", ""))
    rrdata = str(rrdata).strip()
    rrdata_unquoted = rrdata.strip("\"")

    if rtype == "txt" and rhost == host:
        if not token or rrdata == token or rrdata_unquoted == token:
            rid = r.get("id")
            if rid is not None:
                print(rid)
PY
}

delete_txt_record() {
  host="$1"
  token="${2:-}"

  log "Searching TXT records for host=$host token=${token:-none}"

  records_json="$(api_get_records)"
  log "easyDNS records fetched"

  ids="$(find_txt_record_ids "$host" "$token" "$records_json" || true)"

  if [ -z "$ids" ] && [ -n "$token" ]; then
    log "No TXT record found with exact token. Searching again by host only."
    ids="$(find_txt_record_ids "$host" "" "$records_json" || true)"
  fi

  if [ -z "$ids" ]; then
    log "No matching TXT record found for host=$host"
    return 0
  fi

  echo "$ids" | while IFS= read -r id; do
    [ -z "$id" ] && continue
    log "Deleting TXT record host=$host id=$id"
    api_delete_record "$id"
    log "Delete completed for TXT record id=$id"
    sleep 2
  done
}

set_record() {
  challenge_fqdn="$1"
  token="$2"

  host="$(record_to_host "$challenge_fqdn")"

  log "SET requested"
  log "Challenge FQDN=$challenge_fqdn"
  log "Zone=$EASYDNS_DOMAIN"
  log "Host=$host"

  delete_txt_record "$host" "$token" || true

  body="$(json_body_for_add "$host" "$token")"

  log "Adding TXT record host=$host ttl=$EASYDNS_TTL"
  api_add_txt "$body" >> "$LOG_FILE" 2>&1
  log "TXT record added for host=$host"

  log "Waiting ${PROPAGATION_SLEEP}s for DNS propagation"
  sleep "$PROPAGATION_SLEEP"

  log "SET completed"
}

unset_record() {
  challenge_fqdn="$1"
  token="${2:-}"

  host="$(record_to_host "$challenge_fqdn")"

  log "UNSET requested"
  log "Challenge FQDN=$challenge_fqdn"
  log "Zone=$EASYDNS_DOMAIN"
  log "Host=$host"
  log "Token=${token:-none}"

  delete_txt_record "$host" "$token"

  log "UNSET completed"
}

# ------------------------------------------------------------
# Parameter Parsing
# ------------------------------------------------------------
mode="${1:-}"

arg2="${2:-}"
arg3="${3:-}"
arg4="${4:-}"

domain_fqdn=""
challenge_fqdn=""
txt_token=""

log "============================================================"
log "Raw args: mode='${mode:-}' arg2='${arg2:-}' arg3='${arg3:-}' arg4='${arg4:-}'"

case "$mode" in
  set)
    case "$arg2" in
      _acme-challenge*)
        # Fallback-Format:
        # set challenge token
        challenge_fqdn="$arg2"
        txt_token="$arg3"
        ;;
      *)
        # TrueNAS-Format:
        # set domain challenge token
        domain_fqdn="$arg2"
        challenge_fqdn="$arg3"
        txt_token="$arg4"
        ;;
    esac

    [ -n "$challenge_fqdn" ] || fail "Missing challenge FQDN argument"
    [ -n "$txt_token" ] || fail "Missing TXT token argument"

    set_record "$challenge_fqdn" "$txt_token"
    ;;

  unset)
    case "$arg2" in
      _acme-challenge*)
        # Fallback-Format:
        # unset challenge token
        challenge_fqdn="$arg2"
        txt_token="$arg3"
        ;;
      *)
        # TrueNAS-Format:
        # unset domain challenge token
        domain_fqdn="$arg2"
        challenge_fqdn="$arg3"
        txt_token="$arg4"
        ;;
    esac

    [ -n "$challenge_fqdn" ] || fail "Missing challenge FQDN argument"

    unset_record "$challenge_fqdn" "$txt_token"
    ;;

  *)
    fail "Unknown mode '$mode'. Expected 'set' or 'unset'."
    ;;
esac

exit 0

and then use this script when requesting the certificate. The only thing that is not working is the automatic cleanup of the DNS recored, therefore I created this script:'

Code Select Expand

#!/bin/sh

set -eu

LOG_DIR="/mnt/Pool/Scripts/ACME"
LOG_FILE="${LOG_DIR}/cleanup-acme-txt.log"

API_BASE="https://rest.easydns.net"
EASYDNS_DOMAIN="domain.com"

EASYDNS_TOKEN_FILE="/mnt/Pool/Scripts/ACME/easydns.token"
EASYDNS_KEY_FILE="/mnt/Pool/Scripts/ACME/easydns.key"

DRY_RUN=0

if [ "${1:-}" = "--dry-run" ]; then
  DRY_RUN=1
fi

mkdir -p "$LOG_DIR"

log() {
  echo "[$(date '+%F %T')] $*" | tee -a "$LOG_FILE"
}

fail() {
  log "ERROR: $*"
  exit 1
}

[ -f "$EASYDNS_TOKEN_FILE" ] || fail "Token file missing: $EASYDNS_TOKEN_FILE"
[ -f "$EASYDNS_KEY_FILE" ] || fail "Key file missing: $EASYDNS_KEY_FILE"

EASYDNS_TOKEN="$(tr -d '\r\n' < "$EASYDNS_TOKEN_FILE")"
EASYDNS_KEY="$(tr -d '\r\n' < "$EASYDNS_KEY_FILE")"

[ -n "$EASYDNS_TOKEN" ] || fail "Token file is empty"
[ -n "$EASYDNS_KEY" ] || fail "Key file is empty"

TMP_RECORDS="$(mktemp)"
TMP_IDS="$(mktemp)"

cleanup_tmp() {
  rm -f "$TMP_RECORDS" "$TMP_IDS"
}

trap cleanup_tmp EXIT INT TERM

log "============================================================"
log "Starting ACME TXT cleanup for zone: $EASYDNS_DOMAIN"

if [ "$DRY_RUN" -eq 1 ]; then
  log "Mode: DRY-RUN - nothing will be deleted"
else
  log "Mode: DELETE - matching records will be deleted"
fi

to run this once a week to clean up the old records. It is not as smart as centralizing the task from opnsense but it does the job.

[fragrance744]

I ended up setting up ACME in TrueNAS instead.

[Creat]

Did this ever get resolved or anyone find out the actual cause?

My problem is that I did setup the (new) websocket cert push, and I know it worked fine as I swapped over to that cert. Now it needed a renew, and for some reason it failed to push, showing the same symptoms as described here.

I do NOT want to run letsencrypt on the TrueNAS box as well. That's the whole point of having it on the router/firewall.

To be more explicit:
The only normally accessible error in the log is this:

Code Select Expand

AcmeClient: AcmeClient: The shell command returned exit code '1': '/usr/local/sbin/acme.sh ...When manually running that command that fails, I get the same errors the others posted (TrueNAS is not ready. and all that). to my knowledge, the TrueNAS API also doesn't need (or allow) any configuration. I'm currently still on 25.10.3.1, but .4 seems available.

So what gives?

[fragrance744]

midclt, https://github.com/truenas/api_client, is required for using TrueNAS websocket API.
The script may work after installing it.

However, I'll never install any 3rd party software on my firewall.
I'd rather have TrueNAS run the script itself and set a firewall rule to allow the connection.

[Patrick M. Hausen]

I'd rather have TrueNAS run the script itself and set a firewall rule to allow the connection

I'll ask again: why not use SSH with public key authentication to execute the midclt command on TrueNAS from OPNsense?

[Creat]

midclt, https://github.com/truenas/api_client, is required for using TrueNAS websocket API.
The script may work after installing it.

I'm sorry but that makes no sense. This has already worked after I had swapped the automation to the new websocket API, it just stopped working at some point. How could it have worked if that tool is needed?

Also, midclt isn't what you linked on github. You linked to a script that calls `midclt` LOCALLY, which in turn is present by default on any TrueNAS install. The whole point of a (websocket) API is that you don't need to install anything locally to use it. And installing it locally doesn't seem possible (as it supposedly talks to a local truenas install). Even then, there's NO chance I'm installing `midclt` on my firewall as that wouldn't survive a config backp/restore AND this is the firs thing the GitHub page says: Software in ALPHA state, highly experimental.

[Rene78]

I started this topic from a genuine interest in getting the whole certification chain "secure". Not a rigid requirement as I am in a happy place, I trust the network behind my OPNSense. Therefore I am happy to do end the "correct" certificate chain at the firewall and have self-signed stuff behind my reverse-proxy. this works fine and achieves the goal of having the SSL checks and balances on the internet correct (also in the browser), all the way to the backend services.

midclt, https://github.com/truenas/api_client, is required for using TrueNAS websocket API.
The script may work after installing it.

Also, midclt isn't what you linked on github. You linked to a script that calls `midclt` LOCALLY, which in turn is present by default on any TrueNAS install.

Now, this is incorrect. fragrance744 is correct. midctl is designed for remote use. midcli (t=i) (https://github.com/truenas/midcli) is local tool and in ALPHA state. Single letter difference, both on GitHub.

I'll ask again: why not use SSH with public key authentication to execute the midclt command on TrueNAS from OPNsense?

Personally, I am unable to get this working (maybe I can spin up my Claude AI... ;)). While midclt is installed by default on TrueNAS, it should be possible to also use the Websocket API from plugin code, right? This is imho the most elegant solution its per-design way of doing thing.

Bummer is that I am not a developer, am not sure if my aforementioned ideal solution is possible, but let's play nice to each other. I am very happy with the community to build the plugins with all the cool options that OPNSense is bringing to the table. I would contribute if able, and vibe-coding something without understanding its full working is something that creates shortcircuits in my head.

Bottom-line and summarizing there are five viable options:
1) Do not complete the cert chain all the way to the backend services, stop it at the reverse proxy and use selfsigned from there to the backend (my current solution, fine for my home network)
2) Use the SSH option that Patrick suggests and run midclt on the TrueNAS box locally. Basically this also uses the Websocket API, but only through the localhost. The API on TrueNAS apparently listens on localhost as well.
3) Use the ACME client automation with the older implementation. This works for TrueNAS 25.x.x and older.
4) Install the midclt on the OPNSense box. Hopefully this would make ACME client plugin automation (Websocket API) work again and use that. If not, this would require (shell) coding eveything on the OPNSense box. I would say "revert to 2)"
5) Update the ACME client plugin and code something that does not rely on midclt. Basically, remove the apparent bug from this piece of plugin.