Caddy Proxy - Install and Use

[guest15389]

For my use case, I found Caddy to be a much simpler solution than the standard plugins as it supports oAuth authentication and automatic cert renewal via LetsEncrypt.

It uses a much simpler configuration than NGINX and for me, just works better with less hassle and it includes a FreeBSD binary with everything you want already.

I install wget and just grab the binary from their site with the plugins that I need. It allows for flexibility in terms of what plugins that you need to build a minimal install.

Code: [Select]

pkg install wget

cd
mkdir caddy
cd caddy
wget -O caddy.tar.gz "https://caddyserver.com/download/freebsd/amd64?plugins=http.cache,http.cgi,http.jwt,http.login,tls.dns.cloudflare&license=personal&telemetry=on"
tar zxvf caddy.tar.gz

cp -rp caddy /usr/local/bin

cd init/freebsd
cp caddy /usr/local/etc/rc.d


For me, I use CloudFlare DNS as my cert verification as CloudFlare is free and handles DNS rather than opening other ports for web server validation.

I use Google oAuth with the login/JWT plugins for my login verification as it works wonderfully easy.

For startup, I just added a line to my /etc/rc.conf

Code: [Select]

echo "caddy_enable=YES" >> /etc/rc.conf
cat /etc/rc.conf

# Validate the line is there
netdata_enable=YES
caddy_enable=YES
Make any changes you need for your caddy start up script in /usr/local/etc/rc.d/caddy

Code: [Select]

vi /usr/local/etc/rc.d/caddy

root@phoenix:/usr/local/etc/rc.d # cat caddy
#!/bin/sh
#
# PROVIDE: caddy
# REQUIRE: networking
# KEYWORD: shutdown

#
# Add the following lines to /etc/rc.conf to enable caddy:
# caddy_enable (bool):        Set to "NO" by default.
#                             Set it to "YES" to enable caddy
#
# caddy_cert_email (str):     Set to "" by default.
#                             Defines the SSL certificate issuer email. By providing an
#                             email address you automatically agree to letsencrypt.org's
#                             general terms and conditions
#
# caddy_bin_path (str):       Set to "/usr/local/bin/caddy" by default.
#                             Provides the path to the caddy server executable
#
# caddy_cpu (str):            Set to "99%" by default.
#                             Configures, how much CPU capacity caddy may gain
#
# caddy_config_path (str):    Set to "/usr/local/www/Caddyfile" by default.
#                             Defines the path for the configuration file caddy will load on boot
#
# caddy_user (str):           Set to "root" by default.
#                             Defines the user that caddy will run on
#
# caddy_group (str):        Set to "wheel" by default.
#                             Defines the group that caddy files will be attached to
#
# caddy_logfile (str)       Set to "/var/log/caddy.log" by default.
#       Defines where the process log file is written, this is not a web access log
#
# caddy_env (str)       Set to "" by default.
#       This allows environment variable to be set that may be required, for example when using "DNS Challenge" account credentials are required.
#       e.g. (in your rc.conf)   caddy_env="CLOUDFLARE_EMAIL=me@domain.com CLOUDFLARE_API_KEY=my_api_key"
#

. /etc/rc.subr

caddy_env="CLOUDFLARE_API_KEY=someAPIKEY CLOUDFLARE_EMAIL=someone@gmail.com"

name="caddy"
rcvar="${name}_enable"

load_rc_config ${name}

: ${caddy_enable:="NO"}
: ${caddy_cert_email="someone@gmail.com"}
: ${caddy_bin_path="/usr/local/bin/caddy"}
: ${caddy_cpu="99%"} # was a bug for me that caused a crash within jails
: ${caddy_config_path="/var/lib/caddy/Caddyfile"}
: ${caddy_logfile="/var/lib/caddy/logs/caddy.log"}
: ${caddy_user="root"}
: ${caddy_group="wheel"}

if [ "$caddy_cert_email" = "" ]
then
    echo "rc variable \$caddy_cert_email is not set. Please provide a valid SSL certificate issuer email."
    exit 1
fi

pidfile="/var/run/${name}.pid"
procname="${caddy_bin_path}" #enabled builtin pid checking for start / stop
command="/usr/sbin/daemon"
command_args="-p ${pidfile} /usr/bin/env ${caddy_env} ${procname} -cpu ${caddy_cpu} -log stdout -conf ${caddy_config_path} -agree -email ${caddy_cert_email} < /dev/null >> ${caddy_logfile} 2>&1"

start_precmd="caddy_startprecmd"

caddy_startprecmd()
{
if [ ! -e "${pidfile}" ]; then
install -o "${caddy_user}" -g "${caddy_group}" "/dev/null" "${pidfile}"
fi

if [ ! -e "${caddy_logfile}" ]; then
install -o "${caddy_user}" -g "${caddy_group}" "/dev/null" "${caddy_logfile}"
fi
}

required_files="${caddy_config_path}"

run_rc_command "$1"
I could probably make this a little cleaner, but didn't see the need as I just left all my caddy stuff in /var/lib/caddy.

Code: [Select]

root@phoenix:/var/lib/caddy # ls
Caddyfile logs ssl

The Caddy config is very simple.

My plex part looks like:

Code: [Select]

# Plex Server
plex.somewhere.us {
gzip
timeouts none
log /opt/caddy/logs/plex.log
tls {
        dns cloudflare
}
proxy / 127.0.0.1:32400 {
        transparent
        websocket
    }
}

Here is an example of the Google Auth as I forward my ruTorrent to a local server.

Code: [Select]

https://rutorrent.domain.us {
gzip
log /var/lib/caddy/logs/rutorrent.log
tls {
        dns cloudflare
}
jwt {
    path /
    redirect /login?backTo={rewrite_uri}
    except /favicon.ico
    allow email admin@domain.us
    log-file /var/lib/caddy/logs/jwt.log
    log-level info
}

  login {
    redirect_check_referer false
    google client_id=clientidhere,client_secret=clientsecret,scope=h
ttps://www.googleapis.com/auth/userinfo.email
    jwt_expiry 168h
    cookie_expiry 2400h
  }
proxy / http://192.168.1.30 {
        transparent
        }
}
So that pops a simple Sign In box that Google authenticates me into my hosted websites.

[jcdick1]

I apologize for necro'ing this, but I didn't get an answer to my questions regarding it in a new thread.

In your second code snippet, you list the items that go into /etc/rc.conf but my OPNsense doesn't have a /etc/rc.conf and I wasn't sure where else that might go.  Do I just make one that contains only the items you have in your code snippet?

Thanks!  And again, I apologize.

[guest15389]

Yep. You can just make the file with that one line in it and that would be fine.

By default, you are correct, it does not exist.

[mimugmail]

echo "caddy_enable=YES" >> /etc/rc.conf.d/caddy

should also do the trick

[jcdick1]

Since I'm opening 80 and 443 directly to the router for Caddy to capture, I would guess I should configure things so the GUI only listens on the internal LAN interface instead of "All" since it is also on 443?

Also, I'm not super clear with FreeBSD how to get the service actually registered so it can be started without rebooting the router.  I tried sysrc caddy_enable=YES but when I try "service caddy start" it just tells me caddy doesn't exist.  FreeBSD is definitely a bit different than Linux.

Thanks so much for the clarification you've provided so far to this newcomer.

[mimugmail]

I for myself always change it to port 4444 so 443 is free to use
Can you check if there is a file called caddy in /usr/local/etc/rc.d/?

[jcdick1]

Yes, I copied the file from <extraction location>/init/freebsd into /usr/local/etc/rc.d/ per the guide, modifying it for my Cloudflare account info environment variables.

[jcdick1]

I did finally manage to at least get the caddy command to kick off.  I had to go in and change the /usr/local/etc/rc.d/caddy init script to executable.

Now I have to figure out why it won't stay running and how to get a readable log file explaining what's killing it.  The log file entry in the caddy init script in the example provided in this thread

Code: [Select]

: ${caddy_logfile="/var/lib/caddy/logs/caddy.log"}
does not seem to create a log file.  Perhaps its a version difference and this variable was deprecated, as the comments section above these variables in my version does not include that entry.  Instead it is:

Code: [Select]

# caddy_syslog_facility (str) Set to "local7" by default.
#                             Defines the syslog facility used to log output from the caddy process.
#                             This is NOT the web access log.
#
# caddy_syslog_level (str)    Set to "notice" by default.
#                             Defines the syslog level used to log output from the caddy process.
#                             This is NOT the web access log.

...

: ${caddy_syslog_facility="local7"}
: ${caddy_syslog_level="notice"}


However, there is no useful information regarding Caddy's failure in system.log or any other log I can see.  I am not getting a caddy.log file generated anywhere.

[guest15389]

If you check my example above, you need to use the log file directive in your Caddyfile and write the log somewhere.

[jcdick1]

I do have a

Code: [Select]

log /var/lib/caddy/caddyservice.log
in my /var/lib/caddy/Caddyfile, as well as the aforementioned

Code: [Select]

: ${caddy_logfile="/var/lib/caddy/caddy.log"}
in the init script.  There are no log files being generated that I can see in either location.  Nor anything in system logs in /var/log to indicate what's killing the process.

I guess I try running it directly from the command line, with various options to see if it logs its failure to the console.

[guest15389]

Run it from the command line and see what the error is.

[jcdick1]

Okay, running from command line, it appears it is having an issue with cloudflare and DNS.  I find only a few results on Google for the message I get, and they seem to indicate that it is the result of my "split DNS," because I use the same domain.tld for all my machines locally as I am trying to resolve externally.  So now its down to configuring OPNsense properly, I think.

Code: [Select]

acme: error cleaning up: cloudflare: failed to find zone domain.tld.: ListZonesContext command failed: error from makeRequest: HTTP status 400: content "{\"success\":false,\"errors\":[{\"code\":6003,\"message\":\"Invalid request headers\",\"error_chain\":[{\"code\":6102,\"message\":\"Invalid format for X-Auth-Email header\"},{\"code\":6103,\"message\":\"Invalid format for X-Auth-Key header\"}]}],\"messages\":[],\"result\":null}"
followed a couple lines later by

Code: [Select]

acme: error presenting token: cloudflare: failed to find zone domain.tld.: ListZonesContext command failed: error from makeRequest: HTTP status 400: content "{\"success\":false,\"errors\":[{\"code\":6003,\"message\":\"Invalid request headers\",\"error_chain\":[{\"code\":6102,\"message\":\"Invalid format for X-Auth-Email header\"},{\"code\":6103,\"message\":\"Invalid format for X-Auth-Key header\"}]}],\"messages\":[],\"result\":null}"

[jcdick1]

I removed some extraneous quotation marks and everything seemed to go.  Running it from CLI on its own, the console output said a certificate had been issued. Running "service caddy status" returns its PID consistently, so now I think it's my firewall rules. 

I've put port 443 (HTTPS) into my WAN firewall rules with a source of "WAN net" and destination of "This Firewall" but I still get a timeout.  I had CloudFlare's proxy service both enabled and disabled, to no effect.  I can't get to my services.

If I forward the port on my router and use my WAN IP:port the login comes up immediately.  If I try to go through HTTPS and the domain name used in the reverse proxy, the connection times out.  I'm still trying to find a log file to parse separate from the one specified in the Caddyfile, as that seems to only log what I would call "superficial" events such as loading this or that image file into the proxy cache, not system type events, like "Yes, I've contacted the upstream DNS, certificate issued/loaded" except what I got when running Caddy directly from command line.