How to use a separate computer as gateway

[randomwalk]

I have an interesting problem and would appreciate everyone's ideas.

I would like to use a separate computer (VM) on the network as a gateway and I have it mostly working, except two issues.  Specifically, I have a Ubuntu VM (192.168.1.10) on the LAN network (192.168.1.1/24) with Mullvad's app installed.  I have configured the VM to act as a gateway where all incoming traffic gets redirected out via Mullvad's Wireguard interface.  I prefer this over the built-in Wireguard plugin in OPNsense because the Mullvad app has additional features such as periodic rotation of the private key, rotation of the VPN server, quantum resistant encryption, etc.

In OPNsense, I manually add this VM (192.168.1.10) as a gateway.  Using firewall rules, I selectively direct some LAN traffic to use this VM as the gateway and can confirm that it works correctly.  For example, when using this gateway, the public IP address of LAN devices is the VPN server.

So the LAN traffic goes like this:  LAN device (192.168.1.x) --> Ubuntu VM (192.168.1.10) --> out via Mullvad interface on the VM --> OPNsense gateway (192.168.1.1) --> internet.

So this is all great, except there are at least two problems.

1)  I cannot figure out how to get Unbound to use this manually added gateway to send out DNS queries.  If I use the built-in OPNsense VPN functionality, then I can assign the VPN as an interface, which would allow me to configure Unbound to use the VPN as the outgoing interface.  However, my manually added VM (192.168.1.10) cannot be assigned an interface, so I'm not sure if there is some way around this problem.  Ideally, I would like to have DNS queries go through the VPN gateway so there is no DNS leak.

2)  I am not sure how to get devices on other subnets to use the VM as gateway.  For example, let's say I have a guest subnet (192.168.2.1/24).  If I use firewall rules to direct traffic on Guest Net to use 192.168.1.10 as the gateway, it does not work.  I think this might be because the default gateway on Guest Net is 192.168.2.1 and I don't think Guest Net devices can access 192.168.1.10.  I have tried adding a firewall rule to allow Guest Net devices pass traffic to 192.168.1.10, but that does not seem to solve the problem.

Any ideas would be greatly appreciated!

[bartjsmit]

Why not run another DNS server (such as pi-hole) and use that as a resolver for OPNsense, and therefore unbound.

The other networks could have an alternative default gateway as well. Can you add your Ubuntu to more VLAN's and set it as the DG there?

Bart...

[randomwalk]

Why not run another DNS server (such as pi-hole) and use that as a resolver for OPNsense, and therefore unbound.

The other networks could have an alternative default gateway as well. Can you add your Ubuntu to more VLAN's and set it as the DG there?

Bart...

Very interesting idea about adding a DNS server! It would add a hop in the lookup chain, but it should work. Thank you!

For the second point, how would I add the Ubuntu VM (192.168.1.10) to other subnets? I can certainly put that VM in its own VLAN (like 192.168.3.1/24), but I don't see how that would help. And I'm not aware of a way to have multiple IP addresses for the same computer (for example, I am not aware of a way for the VM being both 192.168.1.10 and 192.168.2.10 at the same time so that it can be the gateway for both 192.168.1.1/24 and 192.168.2.1/24).

[bartjsmit]

how would I add the Ubuntu VM (192.168.1.10) to other subnets?

Make it multi-homed, i.e. give it additional virtual network interfaces

[Maurice]

So the LAN traffic goes like this:  LAN device (192.168.1.x) --> Ubuntu VM (192.168.1.10) --> out via Mullvad interface on the VM --> OPNsense gateway (192.168.1.1) --> internet.

Nope, it doesn't. If you use policy-based routing on OPNsense to achieve this, the LAN devices will keep using OPNsense as their default gateway, which then forwards the packets:

LAN device --> OPNsense --> Ubuntu VM --> OPNsense --> Internet

To avoid this, you can assign the Ubuntu VM as the default gateway via DHCP.

I cannot figure out how to get Unbound to use this manually added gateway to send out DNS queries.

If you use Unbound as a recursive resolver, you could create an outbound firewall rule on the WAN interface. Destination port: TCP/UDP 53, gateway: Ubuntu VM.

I am not sure how to get devices on other subnets to use the VM as gateway.

Either multi-home the Ubuntu VM and use DHCP to assign it as the default gateway to the devices in the guest net (like bartjsmit suggested). Or keep using OPNsense as the gateway and create a route on the Ubuntu VM which routes your guest subnet back to OPNsense.

Cheers
Maurice

[randomwalk]

So the LAN traffic goes like this:  LAN device (192.168.1.x) --> Ubuntu VM (192.168.1.10) --> out via Mullvad interface on the VM --> OPNsense gateway (192.168.1.1) --> internet.

Nope, it doesn't. If you use policy-based routing on OPNsense to achieve this, the LAN devices will keep using OPNsense as their default gateway, which then forwards the packets:

LAN device --> OPNsense --> Ubuntu VM --> OPNsense --> Internet

To avoid this, you can assign the Ubuntu VM as the default gateway via DHCP.

[/quote]

Making the VM multi-homed makes sense and it would be easy to add more virtual NICs.

I had not realized the above flow of traffic, which makes sense since if the traffic does not first hit OPNsense, then it can't do policy-based routing. I was thinking that because the VM and LAN devices are in the same subnet (192.168.1.1/24), then traffic between them wouldn't go through OPNsense.

I have two concerns about making the VM the default gateway (via DHCP or otherwise).

First, making the VM the default gateway would mean that none of the firewall rules or policy-based routing in OPNsense would have any effect, right? Since the traffic doesn't go to OPNsense first?

Second, since the VM currently forwards all incoming traffic on the LAN interface to wg-mullvad interface, LAN devices would not be able to access local subnets (devices in both the LAN and Guest nets) unless I somehow add iptable rules on the VM to direct "local" traffic to OPNsense. Effectively, I have to make the VM a router too?

It seems hard to decide whether making the VM the default gateway (so saving one hop to OPNsense for external traffic) is more efficient than the current setup where traffic across local subnets goes only through OPNsense without going through the VM, but external traffic requires an additional hop?  Just trying to understand the pros and cons of having the VM be the default gateway.

[Maurice]

I was thinking that because the VM and LAN devices are in the same subnet (192.168.1.1/24), then traffic between them wouldn't go through OPNsense.

It won't go through OPNsense if the destination is the VM itself, e. g. when accessing it via SSH. But for Internet-bound traffic, the destination addresses will be outside the local subnet, so the LAN devices have no choice but using their default gateway - OPNsense.

First, making the VM the default gateway would mean that none of the firewall rules or policy-based routing in OPNsense would have any effect, right? Since the traffic doesn't go to OPNsense first?

That's correct.

Second, since the VM currently forwards all incoming traffic on the LAN interface to wg-mullvad interface, LAN devices would not be able to access local subnets (devices in both the LAN and Guest nets) unless I somehow add iptable rules on the VM to direct "local" traffic to OPNsense.

LAN devices would still be able to access other devices in the same LAN (including OPNsense) because they are in the same subnet. If the VM is multi-homed, accessing other LANs would require rules on the VM to control traffic between interfaces.

Effectively, I have to make the VM a router too?

Correct. But one way or another, you'll have to do this anyway.

Just trying to understand the pros and cons of having the VM be the default gateway.

I would keep OPNsense as the default gateway and create a dedicated LAN for the WireGuard VM only. Then use policy-based routing on OPNsense and possibly static routes on the VM to direct traffic flow.