Wireguard as an alternative upstream for some VM (no LB, no failover)

[benpi]

Hi,

What i try to do ?
Use a second upstream with Wireguard and only for several VM (identified by ip), all other VM have to use the default upstream.

tldr : wrong route/upstream when VM answer to incomming tcp request

After several days, i still have some problem and i hope to solve it here.
I have a better knowledge of linux (iptables) than freebsd/opnsense then maybe i did a dummy mistake.

Detail
As describe by this picture (network reduced to the minimum), i have one vm witch has to go thourgh the Alternative upstream.

In Green : a command on VM as like as wget http://example.com
- 1 : packet goes to Router
- 2 : packet forwarded to http://example.com through Wireguard (and will be masqueraded)
- 3 : answer from http://example.com goes to Router
- 4 : Router forwarded it to VM

tcpdump on Router :
On vtnet2 : 1 and 4 are visibles
On wg0 : 2 and 3 are visibles

==> It works

In Orange : an incomming connexion on TCP:12345 (num port doesn't matter, just for the explanation)
- 1 : packet goes from internet trhough Wirewuard (DNAT = VM ip)
- 2 : packet forwarded to VM
- 3 : answer from VM
- 4 : Router forwarded it to default upstream instead of wireguard

tcpdump on Router :
On wg0 : only 1 is visible
On vtnet2 : 2 and 3 are visibles
On vtnet1 (default upstream) : 4 is visible

==> why 4 is on vtnet1 instead of wg0 ?

Configuration

All in IPv4, no IPv6
Additionnal comments are in blue

# Wireguard

- Endpoint

Enabled	Checked
Name	VpnSrv
Public Key	xxxxxxxxxxxx
Shared Secret	xxxxxxxxxxxx
Allowed IPs	0.0.0.0/0 (works with this value, doesn't work if more precision)
Endpoint Adress	<IP of the remote server>
Endpoint Port	<Port of the vpn>
Keepalive	25 (but useless with monitor IP on gateway)

- Local

Enabled	Checked
Name	VpnClt
Instance	0
Public Key	xxxxxxxxxxxx
Private Secret	xxxxxxxxxxxx
Listen Port	<Port of the vpn> (same as server, doesn't matter i think)
MTU	1350 (less than value found with ping)
DNS Server	<IP of DNS>
Tunnel Address	1.66.66.2/24
Peers	VpnSrv
Disable Routes	Unchecked (because route only for several VM)
Gateway	1.66.66.1[/color]

# Gateways

Name	Interface	Protocol	Priority	Gateway	Monitor IP	RTT	RTTd	Loss	Status
WAN_GWv4	WAN_I	IPv4	200(upstream)	192.168.0.1	192.168.0.1	1.3ms	0.5ms	0.0%	Online
WireGuard_GWv4	WireGuard_I	IPv4	255	1.66.66.1	1.66.66.1	59.4ms	0.4ms	0.0%	Online

# Rules

- vtnet2 (no reply-to found)

Code: [Select]

# VM route to the VPN
pass in log quick on vtnet2 route-to (wg0 10.66.66.1) inet from <IP_of_my_VM> to (wg0:network) flags S/SA keep state label "aab85546d68e895405a9c85302113012"
pass in log quick on vtnet2 route-to (wg0 10.66.66.1) inet from <IP_of_my_VM> to ! <IPGROUP_RF1918> flags S/SA keep state label "de1cec96bcfb256207074d627c8fddef"

# VM default drop (if one or the both previous rules are deactivated)
block drop in log quick on vtnet2 inet from <IP_of_my_VM> to ! <IPGROUP_RF1918> label "f89b10f01e32d152333a332bb4769cbe"

# HA proxy "loop"
pass in log quick on vtnet2 inet proto tcp from <IP_of_my_VM> to <IP_rout2dmz1_on_RTG> port = http flags S/SA keep state label "c21f2bf8abb986c9a6c20b4e47a0016e"
pass in log quick on vtnet2 inet proto tcp from <IP_of_my_VM> to <IP_rout2dmz1_on_RTG> port = https flags S/SA keep state label "c21f2bf8abb986c9a6c20b4e47a0016e"

- wg0

Code: [Select]

# accept icmp
pass in log quick on wg0 inet proto icmp from any to (wg0) keep state label "944fd4137aef7e8321b6f17aedf03766"
pass in log quick on wg0 inet proto icmp from any to <IP_of_my_VM> keep state label "a8b41fd23302fb0859a894db911728b0"

# allow incomming tcp/udp to my VM
pass in log quick on wg0 inet proto tcp from ! <IPGROUP_RF1918> to <IP_of_my_VM> flags S/SA keep state label "b37665a4b2903815f66d0d642aad66b0"
pass in log quick on wg0 inet proto udp from ! <IPGROUP_RF1918> to <IP_of_my_VM> keep state label "b37665a4b2903815f66d0d642aad66b0"


Questions :
- Why i can't found "reply-to" rules ? (any change with disable reply-to : checked/unchecked )
- How to define this alternative upstream as the only upstream for some VM and keep the other upstream for all other VM ?


I can give you more information if needed.
Thank's for all helps already given through IRC and thank you for your helps too.

PS : sorry for my bad knowledge of english