Show Posts
1
20.7 Legacy Series / Wireguard as an alternative upstream for some VM (no LB, no failover)
« on: August 13, 2020, 12:47:38 pm »
Hi,
What i try to do ?
Use a second upstream with Wireguard and only for several VM (identified by ip), all other VM have to use the default upstream.
tldr : wrong route/upstream when VM answer to incomming tcp request
After several days, i still have some problem and i hope to solve it here.
I have a better knowledge of linux (iptables) than freebsd/opnsense then maybe i did a dummy mistake.
Detail
As describe by this picture (network reduced to the minimum), i have one vm witch has to go thourgh the Alternative upstream.
In Green : a command on VM as like as wget http://example.com
- 1 : packet goes to Router
- 2 : packet forwarded to http://example.com through Wireguard (and will be masqueraded)
- 3 : answer from http://example.com goes to Router
- 4 : Router forwarded it to VM
tcpdump on Router :
On vtnet2 : 1 and 4 are visibles
On wg0 : 2 and 3 are visibles
==> It works
In Orange : an incomming connexion on TCP:12345 (num port doesn't matter, just for the explanation)
- 1 : packet goes from internet trhough Wirewuard (DNAT = VM ip)
- 2 : packet forwarded to VM
- 3 : answer from VM
- 4 : Router forwarded it to default upstream instead of wireguard
tcpdump on Router :
On wg0 : only 1 is visible
On vtnet2 : 2 and 3 are visibles
On vtnet1 (default upstream) : 4 is visible
==> why 4 is on vtnet1 instead of wg0 ?
Configuration
All in IPv4, no IPv6
Additionnal comments are in blue
# Wireguard
- Endpoint
- Local
# Gateways
# Rules
- vtnet2 (no reply-to found)
- wg0
Questions :
- Why i can't found "reply-to" rules ? (any change with disable reply-to : checked/unchecked )
- How to define this alternative upstream as the only upstream for some VM and keep the other upstream for all other VM ?
I can give you more information if needed.
Thank's for all helps already given through IRC and thank you for your helps too.
PS : sorry for my bad knowledge of english
What i try to do ?
Use a second upstream with Wireguard and only for several VM (identified by ip), all other VM have to use the default upstream.
tldr : wrong route/upstream when VM answer to incomming tcp request
After several days, i still have some problem and i hope to solve it here.
I have a better knowledge of linux (iptables) than freebsd/opnsense then maybe i did a dummy mistake.
Detail
As describe by this picture (network reduced to the minimum), i have one vm witch has to go thourgh the Alternative upstream.
In Green : a command on VM as like as wget http://example.com
- 1 : packet goes to Router
- 2 : packet forwarded to http://example.com through Wireguard (and will be masqueraded)
- 3 : answer from http://example.com goes to Router
- 4 : Router forwarded it to VM
tcpdump on Router :
On vtnet2 : 1 and 4 are visibles
On wg0 : 2 and 3 are visibles
==> It works
In Orange : an incomming connexion on TCP:12345 (num port doesn't matter, just for the explanation)
- 1 : packet goes from internet trhough Wirewuard (DNAT = VM ip)
- 2 : packet forwarded to VM
- 3 : answer from VM
- 4 : Router forwarded it to default upstream instead of wireguard
tcpdump on Router :
On wg0 : only 1 is visible
On vtnet2 : 2 and 3 are visibles
On vtnet1 (default upstream) : 4 is visible
==> why 4 is on vtnet1 instead of wg0 ?
Configuration
All in IPv4, no IPv6
Additionnal comments are in blue
# Wireguard
- Endpoint
| Enabled | Checked |
| Name | VpnSrv |
| Public Key | xxxxxxxxxxxx |
| Shared Secret | xxxxxxxxxxxx |
| Allowed IPs | 0.0.0.0/0 (works with this value, doesn't work if more precision) |
| Endpoint Adress | <IP of the remote server> |
| Endpoint Port | <Port of the vpn> |
| Keepalive | 25 (but useless with monitor IP on gateway) |
- Local
| Enabled | Checked |
| Name | VpnClt |
| Instance | 0 |
| Public Key | xxxxxxxxxxxx |
| Private Secret | xxxxxxxxxxxx |
| Listen Port | <Port of the vpn> (same as server, doesn't matter i think) |
| MTU | 1350 (less than value found with ping) |
| DNS Server | <IP of DNS> |
| Tunnel Address | 1.66.66.2/24 |
| Peers | VpnSrv |
| Disable Routes | Unchecked (because route only for several VM) |
| Gateway | 1.66.66.1[/color] |
# Gateways
| Name | Interface | Protocol | Priority | Gateway | Monitor IP | RTT | RTTd | Loss | Status |
| WAN_GWv4 | WAN_I | IPv4 | 200(upstream) | 192.168.0.1 | 192.168.0.1 | 1.3ms | 0.5ms | 0.0% | Online |
| WireGuard_GWv4 | WireGuard_I | IPv4 | 255 | 1.66.66.1 | 1.66.66.1 | 59.4ms | 0.4ms | 0.0% | Online |
# Rules
- vtnet2 (no reply-to found)
Code: [Select]
# VM route to the VPN
pass in log quick on vtnet2 route-to (wg0 10.66.66.1) inet from <IP_of_my_VM> to (wg0:network) flags S/SA keep state label "aab85546d68e895405a9c85302113012"
pass in log quick on vtnet2 route-to (wg0 10.66.66.1) inet from <IP_of_my_VM> to ! <IPGROUP_RF1918> flags S/SA keep state label "de1cec96bcfb256207074d627c8fddef"
# VM default drop (if one or the both previous rules are deactivated)
block drop in log quick on vtnet2 inet from <IP_of_my_VM> to ! <IPGROUP_RF1918> label "f89b10f01e32d152333a332bb4769cbe"
# HA proxy "loop"
pass in log quick on vtnet2 inet proto tcp from <IP_of_my_VM> to <IP_rout2dmz1_on_RTG> port = http flags S/SA keep state label "c21f2bf8abb986c9a6c20b4e47a0016e"
pass in log quick on vtnet2 inet proto tcp from <IP_of_my_VM> to <IP_rout2dmz1_on_RTG> port = https flags S/SA keep state label "c21f2bf8abb986c9a6c20b4e47a0016e"
- wg0
Code: [Select]
# accept icmp
pass in log quick on wg0 inet proto icmp from any to (wg0) keep state label "944fd4137aef7e8321b6f17aedf03766"
pass in log quick on wg0 inet proto icmp from any to <IP_of_my_VM> keep state label "a8b41fd23302fb0859a894db911728b0"
# allow incomming tcp/udp to my VM
pass in log quick on wg0 inet proto tcp from ! <IPGROUP_RF1918> to <IP_of_my_VM> flags S/SA keep state label "b37665a4b2903815f66d0d642aad66b0"
pass in log quick on wg0 inet proto udp from ! <IPGROUP_RF1918> to <IP_of_my_VM> keep state label "b37665a4b2903815f66d0d642aad66b0"
Questions :
- Why i can't found "reply-to" rules ? (any change with disable reply-to : checked/unchecked )
- How to define this alternative upstream as the only upstream for some VM and keep the other upstream for all other VM ?
I can give you more information if needed.
Thank's for all helps already given through IRC and thank you for your helps too.
PS : sorry for my bad knowledge of english