Messages

Heard back from Zenarmor after the 2.7.2 update... "this got pushed to 2.7.3 because of the urgent fixes in 2.7.2"

I heard back from Zenarmor support.... "Our team is currently reviewing the findings and plans to include a fix in the upcoming maintenance release."

[Update: the "heavy" runs stopped after a reboot, and here's why]

Update: the "heavy" runs stopped after a reboot, and here's why

Correction first: the scheduler isn't a fixed 5-hour timer. ipdrstreamer checks every hour and collects only if 4 hours have passed since the last heartbeat:

Code Select Expand

19:16:12 heartbeat: not due  last_sent=15:18:06 next_in=1m53s
20:16:12 -> collected (took 3m29s)
Because each collection finishes ~3.5 minutes after its check starts, the 4-hour mark always falls just after an hourly check, so it runs every 5th hour. The agent does the same thing with checks every 4 hours, so the two coincide every 20 hours.

After rebooting (for the OPNsense 26.7.8 update), ipdrstreamer and zenarmor-agent both started at 16:16:12. At the next two coincidences the collectors started ~6 seconds apart and finished at the same millisecond:

Code Select Expand

16:19:53.214 heartbeat: collected  took=3m40.688s
16:19:53.214 heartbeat: collected  triggered_by=agent  took=3m35.185s
So the agent's request appears to join the collection already in progress. Only one 30-day aggregation runs, CPU stays around 55-60%, and there were no alerts.

Before the reboot, the 2.7.1 upgrade had started ipdrstreamer and the agent 30 seconds apart (22:14:38 vs 22:15:08), and each overlap ran two separate 30-day aggregations (~4m48s each, 90%+ CPU).

Workaround: a reboot that starts both processes together seems to eliminate the heavy runs. The light 5-hourly collection remains (~3.5 min vs ~0.4 s before 2.7.1), which I've reported to Zenarmor support.

[Update: it's the heartbeat, and 2.7.1 made it ~500x slower and runs it twice]

Update: it's the heartbeat, and 2.7.1 made it ~500x slower and runs it twice

I captured today's predicted heavy run at 14:15 (Monit: CPU 95.4% at 14:15:48). This answers the questions from my last post.

Periodical.log during the spike
I saved every version of the file from 14:13 to 14:23. Nothing unusual runs at the spike: scheduled-service-control, userenrich and aliases run every minute; check-remote-elastic and periodic-license-control run every 5 minutes (14:15 and 14:20); check-health ran at 14:14. So dashboard-cache and the per-minute scheduler aren't the cause. There are also no Elasticsearch transforms configured.

The cause: the ipdrstreamer-v2 heartbeat
grep 'heartbeat: collected' /usr/local/zenarmor/log/active/ipdrstreamer-v2*.log shows:

Before 2.7.1 (cli.log, triggered_by=agent, every 4h):

Code Select Expand

2026-10-25T22:51:54 heartbeat: collected took=405ms
2026-10-26T02:51:54 heartbeat: collected took=354ms
2026-10-26T06:51:54 heartbeat: collected took=416ms

After 2.7.1, two collectors:

Code Select Expand

# ipdrstreamer's own scheduler, every 5h from its start (22:14:38 on Oct 26)
2026-10-28T23:18:10 heartbeat: collected took=3m32.228s
2026-10-29T04:18:02 heartbeat: collected took=3m24.436s
2026-10-29T09:18:05 heartbeat: collected took=3m26.881s
2026-10-29T14:19:27 heartbeat: collected took=4m49.409s

# agent-triggered (cli.log), fired only 4 times since the upgrade
2026-10-27T02:19:07 heartbeat: collected triggered_by=agent took=3m59.286s
2026-10-27T22:19:34 heartbeat: collected triggered_by=agent took=4m26.416s
2026-10-28T18:20:01 heartbeat: collected triggered_by=agent took=4m53.923s
2026-10-29T14:19:55 heartbeat: collected triggered_by=agent took=4m47.296s

What this shows:
- The heartbeat collection went from ~0.4 seconds to 3.5-5 minutes in 2.7.1. During it, Elasticsearch runs the 30-day composite aggregation on conn_all (source IP x day, paged 1,000 buckets) from my last post, saturating 4+ cores.
- 2.7.1 adds a second collector inside ipdrstreamer that runs every 5 hours from ipdrstreamer's start time. That's the :15 / :20 cycle xpking and I see.
- The heavy runs are both collectors running in parallel: today the scheduler started at 14:14:38 and the agent-triggered one at 14:15:08. The agent's 4-hourly check only fires when 4+ hours have passed since the last heartbeat, which with the 5-hourly scheduler lines up once every 20 hours. All four agent-triggered runs match my four 90%+ CPU alerts exactly.

Questions for the Zenarmor team (@sy):
1. What changed in the 2.7.1 heartbeat that takes it from ~0.4 s to ~4 min? It now aggregates 30 days of conn data by source IP and day each time.
2. Is the second (5-hourly) collector in ipdrstreamer intended, and should it share a lock with the agent-triggered one so they can't run at the same time?
3. Can the heartbeat be made incremental, or its interval/lookback configured on Home licenses?

Setup: OPNsense 26.7, Zenarmor 2.7.1, local Elasticsearch 8.11.3, i7-3770 / 32 GB.

Thanks both. xpking's :20 every 5 hours matches my :15 every 5 hours, offset by our different upgrade times.

Periodical.log

On my box this file only ever holds the last minute (about 2 KB). Two samples, Oct 27 07:59 and Oct 29 11:56 (neither at a spike time), look identical:

checking mongodb...
checking mongodb done
dasboard-cache...
CLI dashboard-cache: Caching ...
CLI dashboard-cache: Cached ...
dashboard cache done
elastic-index-check...
elastic-index-check done
done
[2026-10-29T11:56:01][INFO][TEMP] Starting Zenarmor CRON for 2026-10-29 11:56:01

That's followed by Zenarmor's usual 18 internal Cron Check lines, with only scheduled-service-control, userenrich and aliases actually running (all OK). The mongodb / dashboard-cache / elastic-index-check block is always at the top with no timestamps, so it looks like a separate job that rewrites the file each time.

What Elasticsearch is doing during a spike

I logged Elasticsearch's task list and per-thread top every 30 seconds during the 23:15 run last night.

Before (23:13:37): idle. Load was 0.20, and Elasticsearch was running only its own health and monitoring tasks.

From 23:14:39, this search appears:

indices:data/read/search
indices[zenarmor_<id>_conn_all]
"query":{"range":{"start_time":{"gte":1790640000000,"lt":1793232000000}}}
"aggregations":{"daily":{"composite":{"size":1000,
  "sources":[{"identity":{"terms":{"field":"ip_src_saddr.keyword"}}},
             {"day":{"date_histogram":{"field":"start_time","calendar_interval":"day"}}}],
  "after":{"identity":"<ip>","day":...}}}}
The range is Sep 29 to Oct 29 (UTC), exactly 30 days.
It's a composite aggregation by source IP × day, paged 1,000 buckets at a time.
The same query appears in 7 consecutive snapshots (about 3½ minutes), each with a different after key advancing through the source IPs. So it looks like one continuous pass through the whole result set.

At the peak (23:16:10): all 15 of the busiest threads in top -H were Elasticsearch's java, with 13 of them at about 30% each. No other process appeared.

Summary: every 5 hours at :15 since the 2.7.1 upgrade, a 30-day per-device daily aggregation on conn_all runs for about 4 minutes. Every 4th run is heavier (90%+ CPU). There was no such pattern in 11 weeks of CPU history before the upgrade.

Which task sends this query? Is it dashboard-cache? And are the interval or the 30-day window configurable? I'm capturing Periodical.log and the process connecting to port 9200 during my next predicted run at 14:15 today and will post the results.

Update: I exported CPU history from Reporting > Health > System > Processor, and the spikes follow a fixed schedule:

Every 5 hours at :15, counted from when Zenarmor restarted during the upgrade (for me, 22:15 on Oct 26). Today's runs were 03:15, 08:15, 13:15 and 18:15.
Every 4th run is heavier, about once every 20 hours, with CPU at 90%+ for about 4 minutes. The lighter runs peak around 55–60% and didn't trigger my alerts.
Before the upgrade: no such pattern in 11 weeks of history.

The load is application CPU, not network or disk. During one heavy run, Elasticsearch was the top process in Diagnostics > Activity. I going to try to capture Elasticsearch's active tasks during the next predicted runs to identify the job itself and will post the results.

If you check your own Processor graph (5-minute resolution), do you see the same 5-hour rhythm starting from your upgrade time?

Thanks, that's helpful. It sounds like we're seeing the same thing.

My timeline after upgrading at about 22:15 on Oct 26:

First spike at 02:15, about 4 hours after the upgrade
Nothing overnight Oct 27→28
Second spike at 22:15, about 24 hours after the upgrade, which is when the Zenarmor agent restarted.

Elasticsearch itself wasn't restarted by the upgrade, so the trigger seems to be a task inside Zenarmor that sends a heavy job to Elasticsearch.

Could you share the exact times of your spikes relative to your upgrade?

I haven't tried the Restart Zenarmor button yet. I'm capturing Elasticsearch hot threads and tasks during tonight's expected spike (24h after last night's spike), and I'll post the results.

I'll test the restart button after that, when a reboot won't disrupt anything.

Since upgrading to Zenarmor 2.7.1 on Oct 26, I've had two short CPU spikes on my OPNsense box. I've never had one before the upgrade.

When: Oct 27 at 02:15 and 22:15

What: each lasted about 4 minutes, with CPU at 90%+ and the load average around 8–9. It recovered on its own.

Source: the Elasticsearch java process (local Elasticsearch 8.11.3)

Ruled out: network traffic and cron jobs

Setup: OPNsense 26.7, i7-3770 with 32 GB RAM, local Elasticsearch reporting database.

I'm monitoring to see if it repeats.

Has anyone else seen something similar since moving to 2.7.1?

[Major update — 2026-08-23: the 1:1 NAT / BINAT workaround is NOT universally required — it's per-server]

Major update — 2026-08-23: the 1:1 NAT / BINAT workaround is NOT universally required — it's per-server

I want to correct something in my original post that I now believe was wrong, or at least incomplete: I described the BINAT workaround as something you need for any additional simultaneous WireGuard tunnel on a ProtonVPN account. That's not accurate. Whether you need it depends on which specific server you're connecting to — some servers need it, some don't, and there's no way to predict which from the outside.

What I found, testing across multiple tunnels and servers on the same account:

• Some servers reply to ICMP correctly to whatever local address you've configured, with zero NAT tricks needed — real traffic and monitoring both work out of the box.
• Other servers — even using an identical client-side setup — reply to ICMP using the old shared default address (10.2.0.2) regardless of what you've configured locally, and need the BINAT workaround to be reachable for monitoring purposes.
  
• I initially assumed this was caused by reusing a client keypair across more than one server over time — it isn't. I tested this directly: generated a completely fresh keypair, used with exactly one server that had shown the problem, and it exhibited the exact same behavior. A fresh key doesn't avoid it.
  
• Real application traffic (browsing, curl, etc.) works fine either way, on every server I tested — this only affects ICMP-based health monitoring (dpinger, ping, etc.), which is why it's easy to miss if you're not specifically checking.

I honestly don't know why some servers behave this way and others don't — I've emailed ProtonVPN support asking for clarity and will update this post if I get a useful answer. It could be different infrastructure/software versions across the fleet, something account-side, or something else entirely.

Revised recommendation for anyone building multiple simultaneous tunnels: don't build the BINAT/NAT workaround preemptively for every new tunnel. Build the tunnel plainly first (unique local address, standard outbound NAT to "interface address"), then test:

• Check your gateway monitor status (dpinger, or just [tt]ping[/tt] from the firewall using the tunnel's local address as source).
• If ICMP fails but real traffic works, check your firewall's live packet log (Firewall > Log Files > Live View in OPNsense) for the blocked reply — look at the actual destination address.
  
• If it shows the old shared [tt]10.2.0.2[/tt] (or some other unexpected address) instead of your tunnel's real address, then add the 1:1 BINAT rule, translating that address back to your real one.
  
• If you ever change which server a tunnel connects to, re-check — a workaround needed for one server can actively break a different server that replies correctly on its own (learned this the hard way: an active BINAT rule caused a new block once I switched to a server that didn't need it, since its correctly-addressed replies no longer matched the state the BINAT rule expected).
  

In short: test empirically per server, don't assume.

[Quick addendum if you're using or considering Mullvad instead of a second ProtonVPN peer for your rollover tunnel:]

Quick addendum if you're using or considering Mullvad instead of a second ProtonVPN peer for your rollover tunnel:

The NAT gotcha above (Part 6) is a ProtonVPN-specific thing, not a "any second simultaneous WireGuard tunnel" problem. Mullvad's account model is different — each key you register gets its own genuinely unique tunnel address via their API, not a shared placeholder. Confirmed live via tcpdump: replies come back addressed to the real assigned address, not 10.2.0.2. So the whole 1:1 BINAT step (Part 6.1) isn't needed at all with Mullvad.

You do still need the Outbound NAT rule though — that's a separate mechanism from the inbound BINAT fix, and skipping it gives you the same "tunnel's healthy, no LAN internet" symptom for a different reason (learned that one the hard way).

What's different if you're building this with Mullvad instead:

• Generate your own keypair — OPNsense can auto-generate this when you create the instance.
• Register the public key with Mullvad's API. Worth flagging: their own docs' single-step "Authorization: Token" call is now dead (NOT_AUTHENTICATED). Current flow is two steps:
  
  
  Code Select Expand
  curl -s -X POST 'https://api.mullvad.net/auth/v1/token' \
  -H 'Content-Type: application/json' \
  -d '{"account_number": "YOUR_ACCOUNT_NUMBER"}'

curl -s -X POST 'https://api.mullvad.net/accounts/v1/devices' \
  -H "Authorization: Bearer <token from above>" \
  -H 'Content-Type: application/json' \
  -d '{"pubkey":"<your public key>","hijack_dns":false}'
• The response includes your real assigned ipv4_address — use that in the instance's Addresses field.
• Skip Part 6.1 (1:1 BINAT) entirely.
• Still build Part 6.2 (Outbound NAT), just set Translate Source IP to Interface address instead of a literal shared IP — your interface's real address is exactly what Mullvad recognizes.

Everything else above — Gateway field matching, interface assignment, Gateway Group, Monit alerting — applies identically regardless of provider.

[Update - alert email duplication if you're using Gateway Groups + Monit]

Update - alert email duplication if you're using Gateway Groups + Monit

Small follow-up for anyone who builds this and then gets a real (non-test) failover event: if you've got Monit alerting on your individual gateway(s) and you're using Gateway Groups, you'll likely get double emails per event.

Turns out OPNsense native Monit check called "gateway_alert" (script: /usr/local/opnsense/scripts/monit/gateway_alert.php) - separate from any custom per-gateway Monit check you build yourself following a guide like this one - ends up watching the same underlying gateway, so a single real down/up event will fire two separate emails.

A few things worth knowing if you hit this:

• The native "gateway_alert" script is actually more detailed on the down side than a simple custom script - it names the routing group being affected and distinguishes down vs packet-loss vs high-latency triggers.
  
• But it's silent on recovery by design - look at the source and the success path never echoes anything, so Monit just shows a generic "status succeeded (0) -- no output" when a gateway comes back up. No detail at all.
  
• Don't edit that native script directly to fix this - it's core OPNsense code (Copyright Deciso B.V.) and any local change will likely get silently overwritten on your next firmware update.

What I did instead: upgraded my own custom per-gateway alert script to match the native one's down-message detail (names the routing group) AND added a real recovery message with actual delay/loss numbers - something the native script never provides either way. Once my custom script covered everything the native one did plus more, I just disabled the native "gateway_alert" check in Services > Monit > Settings and kept only my own. One informative email per real transition now, in both directions, with zero risk of losing the customization to a future update since it's entirely in a script I own.

Updated script (drop-in replacement for the alert script from earlier in this thread):

Code Select Expand

#!/usr/local/bin/php
<?php
require_once 'config.inc';
require_once 'interfaces.inc';
require_once 'util.inc';
require_once 'plugins.inc.d/dpinger.inc';

$gwname = 'YOUR_GATEWAY_NAME';
$groupname = 'YOUR_GATEWAY_GROUP_NAME';
$gateways_status = dpinger_status();

if (empty($gateways_status[$gwname])) {
   echo "Gateway {$gwname} status could not be determined." . PHP_EOL;
   exit(1);
}

$status = $gateways_status[$gwname]['status'];
$delay = $gateways_status[$gwname]['delay'];
$loss = $gateways_status[$gwname]['loss'];

if (stristr($status, 'down') !== false) {
   echo "MONITOR: {$gwname} is down, removing from routing group {$groupname}." . PHP_EOL;
   exit(1);
}

echo "MONITOR: {$gwname} is up (delay {$delay}, loss {$loss}), rejoining routing group {$groupname}." . PHP_EOL;
exit(0);

Don't disable the native check blindly - check this first: "gateway_alert.php" only loops over gateways that are members of a Gateway Group (look at the source, it walks getGroupsConfig() -> tiers). If your plain WAN gateway is also a member of some other group on your box, disabling this native check would blind you to a raw WAN outage too - don't do that. Check System > Gateways > Group and confirm exactly which gateways are covered before disabling anything. In my case only the two ProtonVPN gateways were in any group, so my main WAN was never covered by this check either way - disabling it cost me nothing I actually had. Your setup may differ.

(Side note if this does apply to you: a dedicated alert for a full WAN-down event is arguably moot anyway, since the box usually can't get an email out over a WAN link that's actually down - so this mostly only matters if you also want alerts for partial degradation like high WAN latency/loss, or a "we're back, here's how long we were down" message after the fact.)

Also worth knowing if you're new to this: Monit only alerts on a state transition, not ongoing status - so if only your primary gateway goes down and your rollover/secondary was healthy the whole time, you won't get any email at all for the secondary. That's expected, not a gap in coverage.

Real-world confirmation, for what it's worth: this whole setup got its first genuine (non-test) workout the night after I built it - primary tunnel dropped for about an hour due to a real upstream VPN provider hiccup, failed over automatically, recovered and failed back automatically, zero manual intervention needed. That's the actual point of building this in the first place, so good to see it hold up outside of deliberate testing.

[ProtonVPN WireGuard Failover with Gateway Groups - plus a shared-address NAT gotcha that will silently break your second tunnel]

ProtonVPN WireGuard Failover with Gateway Groups - plus a shared-address NAT gotcha that will silently break your second tunnel

TL;DR

• Building automatic failover between two simultaneous ProtonVPN WireGuard tunnels on OPNsense using Gateway Groups.
• Every ProtonVPN WireGuard config shares the identical internal address 10.2.0.2, by design, account-wide. Editing the local [Interface] Address on a second tunnel is necessary but not sufficient - ProtonVPN's server keeps replying to 10.2.0.2 regardless, which silently breaks dpinger monitoring on the second tunnel (route looks fine, manual ping/tcpdump looks fine, but gateway status sits at 100% loss forever).
  
• The fix is NAT on both directions (1:1 BINAT inbound + Outbound NAT translating to the literal shared address), not a routing/firewall-rule change.
• Full walkthrough below, plus the failover Gateway Group setup and a real-world tested failover/failback procedure.

Credit where due: the core of the NAT fix is adapted from this pfSense/ProtonVPN writeup by u/FarsightMeercat, which documents the identical problem independently, just for pfSense rather than OPNsense. I only found it after burning a fair amount of time chasing this myself - hoping this post saves the next OPNsense person that search.


Background / prerequisite

This assumes you already have one working ProtonVPN WireGuard tunnel on OPNsense with accurate gateway monitoring (dpinger genuinely reflecting up/down, not silently stuck on a fake-healthy state). If your single-tunnel monitoring isn't already solid, get that right first - a broken foundation makes failover testing impossible to interpret.

The most common way single-tunnel monitoring silently breaks: the WireGuard instance's own Gateway field (VPN > WireGuard > Instances > edit) must exactly match the System Gateway's IP Address field (System > Gateways > Configuration). A mismatch causes OPNsense to find an unrelated pre-existing host route and skip building the real one - no error in the general log, dpinger just reports a fake-healthy "none" status forever. Worth double-checking this even on an existing single-tunnel setup.


Part 1: Create the second WireGuard instance

Generate a second ProtonVPN WireGuard config (different server than your primary tunnel, ideally different physical facility for real redundancy).

Before importing it into OPNsense, edit its [Interface] Address line. Every ProtonVPN config download shows the identical 10.2.0.2/32, regardless of server - confirmed directly with ProtonVPN support: this is intentional, "to avoid static, identifiable patterns," not a per-device assignment. Their own guidance for running multiple tunnels: manually edit each additional config's Address to something unique.

Example addressing scheme:

Code Select Expand

Primary (existing) ......... 10.2.0.2/32
Second tunnel (new) ........ 10.4.0.2/32

VPN > WireGuard > Instances > Add:

Code Select Expand

Public/Private Key .... from the new config
Listen Port ............ any unused port
Addresses ............... 10.4.0.2/32 (edited, NOT the file's original 10.2.0.2/32)
MTU ..................... match your primary (commonly 1420)
DNS Server .............. file also shows a shared 10.2.0.1 - edit this too,
 following the same pattern (10.4.0.1). Note: ProtonVPN
 only confirmed the Address field is shared-by-design;
 I haven't gotten separate confirmation the DNS Server
 field behaves the same way. If DNS fails over the new
 tunnel, fall back to testing the original literal value.
Disable Routes .......... checked - leave routing entirely to the System
 Gateway object, don't let the WireGuard instance
 push its own routes
Gateway ................. LEAVE BLANK FOR NOW

Attach the peer, enable, apply.


Part 2: Set the instance's Gateway field correctly the first time

• With the instance up, confirm a real handshake: wg show <device>
  
• Set the instance's Gateway field. In my case, the working pattern was "one address up from the tunnel address" (10.2.0.2 â†' gateway 10.2.0.3), so I tried the same for the second tunnel: Address 10.4.0.2 â†' Gateway 10.4.0.3. Worked first try.
• Save, apply.

Don't invent an arbitrary placeholder IP here. This exact mistake is what breaks single-tunnel monitoring in the first place (see Background above) - same failure mode applies to a second tunnel.


Part 3: Assign the WireGuard device as an interface

Easy to forget if your primary tunnel already went through this step long ago and you don't remember doing it. A WireGuard instance existing isn't enough on its own - it won't show up in Gateway/NAT/Firewall dropdowns until explicitly assigned.

Interfaces > Assignments: select the new device, add it, enable, set a clear Description (I used VPNClient_Wireguard_Rollover), leave IPv4 Configuration Type as none.

Naming note: the WireGuard instance's own Name field and the assigned interface's Description field are two separate labels for the same tunnel, shown on different screens - Name shows under VPN > WireGuard, Description shows in Gateways/NAT/Firewall dropdowns. Easy to assume they're the same field; they're not.


Part 4: Create the System Gateway object

System > Gateways > Configuration > Add:

Code Select Expand

Interface ........... the interface assigned in Part 3
IP Address ........... EXACTLY the Gateway field value from Part 2
Far Gateway ........... checked
Monitor IP ............ anything distinct from your primary tunnel's
 monitor IP, for easy at-a-glance distinction
Disable Host Route .... unchecked

Save, apply.


Part 5: Verify routing - and expect the next failure

Code Select Expand

route get <monitor IP>Should resolve via the new tunnel interface.

Code Select Expand

tcpdump -ni <new interface> icmpConfirm real ICMP traffic while pinging the monitor IP from the firewall.

At this point, dpinger_status() will likely still show the new gateway as status => down, 100% loss, 0.0ms delay/stddev - even though the route and manual ping both look completely healthy. This tripped me up for a while because it looks like the exact same bug from the Background section, but it isn't - recheck nothing, go straight to Part 6.


Part 6: The actual fix - ProtonVPN's shared address requires NAT, not routing changes

Here's what a tcpdump -ni <new interface> capture during this "down" state actually showed:

Code Select Expand

<new tunnel's real address> > <monitor IP>: ICMP echo request
<monitor IP> > 10.2.0.2: ICMP echo reply

The request leaves correctly from the tunnel's real edited address. The reply comes back addressed to 10.2.0.2 - the original shared ProtonVPN address, not the edited local one. ProtonVPN's server-side infrastructure still associates your registered device/key with 10.2.0.2 internally, regardless of what you configure locally. Since nothing on the box (that dpinger's probe socket is bound to, anyway) owns that address anymore, the reply never reaches the monitor - even though the tunnel itself is completely healthy and passing traffic.

This is a documented ProtonVPN limitation for anyone running 2+ simultaneous WireGuard tunnels from one account, not something fixable via local routing/firewall-rule tweaks, and not fixable by regenerating the peer (ProtonVPN's Peer Generator doesn't let you choose this address).

The fix, adapted from the pfSense writeup linked above - NAT on both directions:

1. Inbound: NAT 1:1 (BINAT), scoped to the new tunnel's interface only

Firewall > NAT > 1:1 > Add:

Code Select Expand

Interface ........ your new tunnel's interface - CRITICAL: must not be
 your primary tunnel's interface
Type .............. BINAT
External network .. 10.2.0.2/32
Source ............ Single host or Network â†' your tunnel's real edited
 address (e.g. 10.4.0.2/32)
Destination ....... any

2. Outbound: Source NAT, translating LAN traffic to the address ProtonVPN actually recognizes

Firewall > NAT > Outbound (Source NAT) > Add:

Code Select Expand

Interface ............. your new tunnel's interface
Source ................ LAN network
Destination ........... any
Translate Source IP ... Single host or Network â†' 10.2.0.2 (literal) - 
 do NOT use "Interface address", which would
 translate to your synthetic edited address,
 which ProtonVPN doesn't recognize as a
 registered device; traffic would go out and
 get silently dropped

Save and apply both. Rule order relative to other outbound rules doesn't matter - pf matches outbound NAT by scoped interface, so a packet leaving via your new tunnel can only match a rule scoped to that same interface.

Restart dpinger and re-check:

Code Select Expand

pluginctl -s dpinger restartThe new gateway should now show status => none with real non-zero delay/stddev and 0.0% loss.


Part 7 (optional but recommended): Floating rule for self-originated traffic

Not the fix for Part 6's issue, but worth having for consistency if your primary tunnel already has one. This affects routing for traffic the firewall generates itself (dpinger's own probes, manual curl/ping from the firewall shell) - separate from Part 6's address translation fix.

Firewall > Rules > Floating > Add:

Code Select Expand

Interface ................. any
Direction .................. In
Action ...................... Pass
Source ....................... new tunnel's interface address alias
Destination ................... new tunnel's interface network alias
Gateway (Source Routing) ...... your new System Gateway


Part 8: Monit alerting + watchdog

Same pattern as any dpinger-based gateway monitoring. A couple of gotchas worth flagging if you haven't already run into them on your primary tunnel:

• Cron's "Minutes" field: "2" means "once per hour at :02," not "every 2 minutes." Use */2.
  
• Cron only runs predefined configd actions - you need a .conf file under /usr/local/opnsense/service/conf/actions.d/ plus a configd restart before the action appears in the Cron job dropdown. Use "service configd restart" - "configctl configd restart" will error with "Action not allowed or missing," since configd can't restart itself through its own action-dispatch mechanism.
  
• Don't trust crontab -l or /etc/crontab to confirm a GUI-managed cron job is actually scheduled - both came back empty for a legitimately-configured, working job in my case. System > Settings > Cron in the GUI is the reliable source of truth. If you want airtight proof of execution (not just that it's configured), temporarily add an unconditional trace line to the top of the script, wait a few cron cycles, and check for timestamped entries.
  
• dpinger doesn't reliably self-clear a stuck "down" state after a real outage resolves - a small watchdog script comparing WireGuard's own handshake freshness against dpinger's reported status, restarting dpinger only when they disagree, fixes this. "pluginctl -s dpinger restart" restarts all dpinger instances on the box (global side effect, not a bug) - worth folding multiple tunnels' checks into one combined watchdog rather than running several independent ones that'll all trigger the same restart anyway.
  

Part 9: Create the Gateway Group

System > Gateways > Group > Add:

Code Select Expand

Gateway Priority ... primary â†' Tier 1, new tunnel â†' Tier 2
Trigger Level ....... Member Down to start (tighten to packet-loss/
 latency thresholds later once you trust it)


Part 10: Point LAN policy routing at the Gateway Group

Firewall > Rules > LAN - find the rule enforcing your VPN gateway for LAN egress (source LAN network, destination !RFC1918_Networks). Change its Gateway field from the single primary gateway to the new Gateway Group. Leave other LAN rules untouched.


Part 11: Test failover and failback properly

• Confirm baseline: both gateways healthy, traffic only on the primary tunnel.
• Disable the primary WireGuard instance (cleaner than physically blocking - exercises the real failure mode dpinger detects).
• Confirm dpinger flips the primary to "down," confirm any Monit alert fires.
• Test from an actual LAN client, not the firewall's own SSH shell. Self-originated firewall traffic follows different routing than forwarded LAN traffic (same category of gotcha as dpinger's own probes needing the Part 7 floating rule) - a curl ifconfig.me run directly on the firewall can silently fall through to raw WAN and give you a false negative. From a real device on your LAN: curl ifconfig.me should show the second tunnel's exit IP.
  
• Re-enable the primary, confirm recovery, re-run the LAN-client test to confirm traffic shifted back.

In my case, this all worked cleanly end to end - failover and failback both confirmed with real client traffic, not just internal monitoring state.


Summary of gotchas that cost the most time

• WireGuard instance Gateway field and System Gateway IP Address field must match exactly, for every tunnel - no placeholders.
  
• ProtonVPN WireGuard configs all share the same internal address (10.2.0.2) by design, account-wide. A local Address edit alone is not sufficient for a second simultaneous tunnel - you need NAT 1:1 (inbound) + Outbound NAT translating to the literal 10.2.0.2 on the new tunnel's interface.
  
• Self-originated firewall traffic (dpinger probes, a manual curl/ping from the SSH shell) follows different routing than real LAN client traffic - always validate failover from an actual LAN device.
• A WireGuard instance's Name field and its assigned interface's Description field are two different labels for the same tunnel on different screens.
• pluginctl -s dpinger restart is global, not per-gateway.

Happy to answer questions if anyone's attempting the same setup - and if anyone's got independent confirmation on the DNS Server field's shared-by-design behavior (see Part 1), I'd love to know either way.

The Monitor IP doesn't need to be your VPN provider's internal DNS address specifically — it just needs to be *some* real host that actually responds to ICMP when routed through that particular tunnel. Mullvad's docs suggesting `10.64.0.1` is just a convenient, known-working example, not a requirement.

Since OPNsense won't let you reuse the same Monitor IP across multiple gateways, the simplest fix is to give each tunnel a different public target — for example:

- Gateway 1: `9.9.9.9` (Quad9)
- Gateway 2: `1.1.1.1` (Cloudflare)
- Gateway 3: `8.8.8.8` (Google)
- Gateway 4: `1.0.0.1` (Cloudflare secondary)

Any of these will work as long as it genuinely answers ping through that specific tunnel — don't just assume it does, confirm it (see below).

One thing worth double-checking based on some fairly extensive troubleshooting I went through recently with a similar multi-gateway WireGuard setup on ProtonVPN: **make sure it's actually the Monitor IP field throwing the duplicate error, and not the Gateway "IP Address" field** (the Far Gateway placeholder). Those are two separate fields on the same System Gateway config screen, and both need to be unique per gateway too — the placeholder "IP Address" also can't be reused across multiple WireGuard gateways on this same box. If you're getting a duplicate error on that field instead of (or in addition to) Monitor IP, you'll need a distinct placeholder value per tunnel as well (e.g. `10.64.1.1`, `10.64.2.1`, `10.64.3.1`, etc. — arbitrary, unused, just needs to be unique).

Also, whichever Monitor IP you land on per gateway, actually verify it before trusting it — this bit me hard on a recent project. Confirm with a packet capture that the ping genuinely traverses the tunnel interface (not silently falling through to WAN):

```
tcpdump -ni <tunnel-interface> icmp
```
while pinging the monitor IP sourced from that tunnel's local address in another window. If nothing shows up on the tunnel interface, the monitor IP isn't actually testing what you think it's testing — regardless of what the dashboard says.

Hope this helps.

I also recommend using Claude as you set it up to help answer questions and troubleshoot.

If you're using a WireGuard client interface as a monitored gateway (for gateway groups, failover, or just health alerting) and your gateway status looks suspiciously *always healthy* — even during a real tunnel outage — check this first. It cost me a long debugging session, and the actual cause turned out to be a simple, easy-to-make configuration mismatch, not a bug.

## The trap

There are **two separate "Gateway" fields**, in two different places, that look related but aren't automatically kept in sync:

1. **VPN > WireGuard > Instances → your instance → "Gateway"** — a field on the WireGuard instance itself.
2. **System > Gateways > Configuration → your gateway object → "IP Address"** — the address you set when creating a monitored gateway for this interface, typically with Far Gateway checked and an arbitrary placeholder IP (since a WireGuard tunnel has no real on-link next-hop).

Nothing in the UI tells you these two need to be the **same address**. If you pick your own arbitrary placeholder for #2 without checking what's already in #1, you end up with two different "gateway" IPs for the same interface — and only the one from #1 actually gets a working route.

## The symptom

- `dpinger_status()` reports the gateway as healthy (`status => none`, `loss => 0.0%`) even when the tunnel is genuinely down.
- Packet capture on the tunnel interface shows *nothing* — your monitor traffic is silently falling through to your WAN default route instead of the tunnel (confirmed via `tcpdump` + checking source-NAT'd traffic on the WAN interface).
- The kernel routing table never gets a route for your Monitor IP via the tunnel, no matter how many times you reload services or reboot.
- No error appears in the general system log — the actual failure is logged in `/var/log/wireguard/latest.log`, not `/var/log/system/latest.log`, and reads something like:
  ```
  route: message indicates error: Invalid argument
  add host <monitor-ip>: gateway <your-placeholder-ip> fib 0: Invalid argument
  ```

## The fix

Check VPN > WireGuard > Instances → your instance → "Gateway" field. Whatever address is there, use **that exact address** as the "IP Address" in your System > Gateways configuration for this interface — don't invent your own placeholder.

In my case, the instance's Gateway field was `10.2.0.3`; I'd used `10.2.0.1` for the System Gateway object. Changing the System Gateway to `10.2.0.3` fixed it completely — no manual static routes needed, confirmed via packet capture that monitor traffic now genuinely traverses the tunnel, and `dpinger_status()` correctly reports real loss/downtime.

## Why this happens (briefly, for the curious)

OPNsense's interface-route-building logic (`system_interface_route()` in `system.inc`) checks whether *any* host route already exists on the device with a link-level gateway before attempting to add its own — but doesn't check whether that existing route is for the specific address it needs. WireGuard's own instance logic already creates a correct route for its own Gateway field; if your System Gateway uses a different address, that unrelated route satisfies the "something already exists" check and the actual route you need is silently never created.

Not a bug, technically — just a very easy trap to fall into, and one the UI doesn't warn you about. Worth checking if your WireGuard gateway monitoring has ever felt "too quiet."

**Update:** if you've fixed the Gateway-field mismatch described above and your gateway *still* seems to never detect a real outage, check one more thing before assuming something's still wrong: dpinger's advanced parameters need to be adjusted too, not just the IP Address.

When I built a second site, I matched the System Gateway's IP Address to the WireGuard instance's Gateway field correctly from the start — no leak, route table looked right — but disabling the peer still didn't flip the status to down within any reasonable time. Turned out the advanced settings on System > Gateways > Configuration (Probe Interval, Time Period, Loss Interval, Data Length, Packet Loss/Latency thresholds, and the **Failover States** checkbox) were left at looser defaults than my known-working gateway. With those loose enough, dpinger just hadn't gathered enough samples yet to trip the alarm — it looked exactly like the routing bug above, but was a completely separate, simpler issue.

**Fix:** when setting up a new WireGuard gateway for monitoring, adjust the *entire* advanced parameter set, not just IP Address / Far Gateway / Monitor IP. In my case, the values that worked for my setup were:

- Failover States: **checked**
- Latency Low/High Threshold: 200 / 500
- Packet Loss Low/High Threshold: 20 / 50
- Probe Interval: 2
- Time Period: *(left blank — uses default)*
- Loss Interval: 4
- Data Length: *(left blank — uses default)*

Notably, leaving Time Period and Data Length **blank** (rather than setting an explicit value) was part of what worked — an explicit value there was one of the things that made detection sluggish on the misconfigured gateway.

TL;DR: "gateway always shows healthy" has (at least) two independent causes — a routing mismatch (see above) and a detection-sensitivity mismatch (this). Worth ruling out both.

I use the os-speedtest-community plugin from mimugmail repository.  I am setting up a new firewall with v26.7, and I cannot find it in the plugins list.  It's alive and working on my older firewall that I upgraded to v26.7 with speedtest already installed.  Thoughts?