Site to Site Wireguard (dynamic ip/VPS) Config Question and Troubleshooting

[bos_fam]

Network

Code Select Expand

    192.168.10.x                    192.168.20.x
    ┌──────────┐                    ┌───────────┐
    │  parents │                    │  sis/bro  │
    │ opnsense │                    │  opnsense │
    └──────────┘                    └───────────┘
            │                              │
        isp router                      isp router
            │                              │
            └──────────┐        ┌──────────┘
                       │        │
                    ┌──▼────────▼──┐
                    │    VPS       │ 192.168.100.x
                    └──────┬───────┘
                           │
                        isp router
                           ▲
                           ║
                    ┌──────▼──────┐
                    │  Primary    │
                    └─────────────┘
                    192.168.50.x

Disclaimers:
- I'm putting this in General since it feels like a fw issue rather than a VPN specific issue.
- I'm aware of tailscale (and use it), but don't want to add another external service and want to learn something as I roll this out

I'm trying to setup a way for my parents and siblings to occaisonally share pdf documents and family photos. The traffic will be bursty when upload/download/viewing, but relatively low traffic most of the time.
A computer on `192.168.10` should be able to access the pdf server on `192.168.50`; no need to support traffic from `192.168.50` back to the other nodes. In general only want to support using ssh, http/https, and icmp (troubleshooting) traversing the tunnels

Connection Info:
- none of the ISPs provide static ipv4 addresses, only one has ipv6 (which seems flaky)
- picked up a low-cost VPS to get static ipv4 (and ipv6) address
- no pass-through mode on any of the ISP boxes
- all "internal" fw ip are `.1`

So Far:
- I've created WG clients and connected to the VPS per various descriptions in the official docs
- the WG status indicates traffic is flowing
- each WG has an associated interface
- the mss clamping is set
- all firewall rules have 'enable logging' selected
- I'm not able to ping the remote wg ip nor the remote fw "inside" ip
- The firewall live view doesn't show block/accept for ping, traceroute, etc

Questions:
- Any big picture comments? (oh, you're just missing <...>)
- With the logging enabled, I'd expected to see firewall rules firing as packets hit the WG interface.
- How to capture packets on the "other side" of the interfaces to help track down if packets are even getting out the 'local' firewall

I can add the interface and firewall details; left out for now to start at the high level and work down to the details

[nicqq]

Network

Code Select Expand

    192.168.10.x                    192.168.20.x
    ┌─────────┐                    ┌───────────┐
    │ parents │                    │  sis/bro  │
    └─────────┘                    └───────────┘
            │                              │
            │                              │
            └──────────┐        ┌──────────┘
                       │        │
                    ┌──▼────────▼──┐
                    │     VPS      │ 192.168.100.x
                    └──────┬───────┘
                           ▲
                           ║
                    ┌──────▼──────┐
                    │   Primary   │
                    └─────────────┘
                    192.168.50.x

Disclaimers:
- I'm putting this in General since it feels like a fw issue rather than a VPN specific issue.
- I'm aware of tailscale (and use it), but don't want to add another external service and want to learn something as I roll this out

I'm trying to setup a way for my parents and siblings to occaisonally share pdf documents and family photos. The traffic will be bursty when upload/download/viewing, but relatively low traffic most of the time.
A computer on `192.168.10` should be able to access the pdf server on `192.168.50`; no need to support traffic from `192.168.50` back to the other nodes. In general only want to support using ssh, http/https, and icmp (troubleshooting) traversing the tunnels

Connection Info:
- none of the ISPs provide static ipv4 addresses, only one has ipv6 (which seems flaky)
- picked up a low-cost VPS to get static ipv4 (and ipv6) address
- no pass-through mode on any of the ISP boxes
- all "internal" fw ip are `.1`

So Far:
- I've created WG clients and connected to the VPS per various descriptions in the official docs
- the WG status indicates traffic is flowing
- each WG has an associated interface
- the mss clamping is set
- all firewall rules have 'enable logging' selected
- I'm not able to ping the remote wg ip nor the remote fw "inside" ip
- The firewall live view doesn't show block/accept for ping, traceroute, etc

Questions:
- Any big picture comments? (oh, you're just missing <...>)
- With the logging enabled, I'd expected to see firewall rules firing as packets hit the WG interface.
- How to capture packets on the "other side" of the interfaces to help track down if packets are even getting out the 'local' firewall

I can add the interface and firewall details; left out for now to start at the high level and work down to the details

whats your wireguard keepalive timer for both (VPS and warriors)?

what if VPS goes down?

[nicqq]

> - I'm not able to ping the remote wg ip nor the remote fw "inside" ip

ups.. regarding your issue.. what's the routing table of peers?

[bos_fam]

FYI, I updated the diagram to reflect the isp router infront of each opnsense instance.

>whats your wireguard keepalive timer for both (VPS and warriors)?

the static ip doesn't have a keep-alive set for the clients (per docs IIRC)
the dynamic ip peers have the a 25s keepalive back to the static server

> what if VPS goes down?

it's not work/critical so I'm taking a weekly snapshot of the VPS and using zfs snapshots in opnsense to help speed up disaster recovery. If needed, I can manually send any pdfs while the VPS is down

> - I'm not able to ping the remote wg ip nor the remote fw "inside" ip

ups.. regarding your issue.. what's the routing table of peers?

for completeness sharing all (with some data dropped re physical interfaces)

static ip opnsense

Code Select Expand

Destination        Gateway            Flags   Nhop#    Mtu            Netif Expire
10.0.100.0/24        link#7             U           9   1420              wg0
10.0.100.1           link#3             UHS        10  16384              lo0
10.0.100.10          link#7             UHS        11   1420              wg0
10.0.100.50          link#7             UHS        11   1420              wg0
192.168.100.0/24     link#2             U           1   1500            vnet0
192.168.100.3        link#3             UHS         3  16384              lo0
192.168.10.0/24      link#7             US         12   1420              wg0
192.168.50.0/24      link#7             US         12   1420              wg0

opnsense .10.1

Code Select Expand

Destination        Gateway            Flags   Nhop#    Mtu            Netif Expire
default              192.168.20.1       UGS         7   1500           vtnet0
10.0.100.0/24        link#7             U           8   1420              wg0
10.0.100.100         link#7             UHS        10   1420              wg0
10.0.100.10          link#3             UHS         9  16384              lo0
127.0.0.1            link#3             UH          2  16384              lo0
192.168.10.0/24      link#2             U           4   1500           vtnet1
192.168.10.1         link#3             UHS         5  16384              lo0

opnsense .50.1

Code Select Expand

default            192.168.1.1        UGS         6   1500             igc1
10.0.100.0/24        link#7             U           7   1420              wg0
10.0.100.50          link#3             UHS         8  16384              lo0
127.0.0.1            link#3             UH          2  16384              lo0
192.168.50.0/24      link#1             U           1   1500             igc0
192.168.50.1         link#3             UHS         3  16384              lo0


It's strange that the static ip opnsense has the wireguard routes, but the dynamic ip opnsense don't have it automatically added.  I thought I missed an "apply" so I restarted and did see this in the dynamic ip wireguard logs.

Code Select Expand

.50.1 - "/usr/local/opnsense/scripts/wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.0.100.50/32' -interface 'wg0'' returned exit code '1', the output was ''"
.10.1 - "/usr/local/opnsense/scripts/wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.0.100.10/32' -interface 'wg0'' returned exit code '1', the output was ''"

The error is not in the static ip opnsense wireguard logs.

So looks like the routes back to the VPS are missing. Should they be added similar to the policy based routing approach? Or as static routes?