Routing issues with far gateway over VPN

[wedge1001]

hi, I hope someone can help me and I'm at the right place.

tldr;
I want to route traffic through a bunch of openSense-Systems and have it exited at a specific opnsense-system.

Code: [Select]

Network client -> opensense 1 -> opensense 2 -> InternetFar-Gateway will not answer me the traffic back.
What did I do wrong?


I have tinkered "a little" bit with my network and added (more or less) every hosting provider I use to this network. They are all connected either via OpenVPN or Wireguard.
An Image of the Network is attached - the doubled routes are handled via BGP - there is no NAT involved with communication between all the networks (except on WAN-Interfaces)

Routing from PCs of each Network to every other network is working fine. So Clients of Network a can ping (and access) Clients from Network b and vice versa.
What I want to do is: allowing Systems of Network a (and [col-or=blue]Network b[/color]) to be routed to opensense 2 and use it's WAN interface for internet traffic.

What I can achieve really easy is letting Network a use opensense 1 as public internet - as well as routing them through an commercial VPN connected this opensense 1;

Code: [Select]

Client of Network a -> opensense 2 -> commercial VPN -> InternetThis works like charm and can easily be achieved with a firewall-rule and force-gateway.

In order to get a gateway for opensense 2 in opensense 1 i did setup a far-gateway with the LAN-IP of opensense 2.
The gateway is up (and surprisingly the monitoring with e.g. 8.8.8.8 ) works and shows acutally online states and rtt.
Unfortunately if I setup a firewall rule to force traffic through this far-gateway traffic will be lost - there won't be an answer at all.
In my Picture the traffic would have to go
Network a -> opensense 1 -> opensense 3 -> opensense 2 -> internet

Do I need to setup my route on every step in between?
Meaning, having to add rules like this:

Code: [Select]

opensense 1 -> opensense 3 gateway
opensense 3 -> opensense 2 gateway
opensense 2 -> internet gateway Thus, creating rules on each system between my source and destination?
In theory every system knows where every other IP is located in the Network, so why do I loose my return route from the far-gateway?

thanks for any help.

[defaultuserfoo]

What actually is a "far" gateway?

[Greelan]

As per the help text, a gateway that exists outside of the interface subnet.

[defaultuserfoo]

Yes, and I don't understand this idea.  I asked about that a while ago.

I would understand "unreachable" or perhaps "disconnected" gateway, but not "far".  Does it mean you need a different interface and a route to reach a "far" gateway?  Or does it need to be renamed?  Or do you need to connect something somehow to something?  What's the idea here?

And I guess you can assume that most gateways exist outside of the subnet of a particular interface.  So what.  Maybe call it an "irrelevant" gateway?  Or, assuming that the very number of gateways that are outside of the subnet of a particular interface may be greater than the number of gateways within the subnet, it's more likely that these gateways are more relevant than a single gateway because there are so many of them, you might want to call it a "foreign" or "outside" gateway?

And what's the furthest gateway?  Going in a straight line or following the curvature of the planet?  Perhaps it's an alien gateway?

So what is it?  What makes you think you could directly reach alien gateways?  You got to go to Halo first :~Z

[Greelan]

Okaaay…

[wedge1001]

okay. I try to simplifly the setup for my question.

I have something like this:

Code: [Select]

WAN / Internet
         .
         |
         |
    .--------------.   private LAN        .--------------.
    | OPNsense1 ----------------------  LAN Clients |
    '----.---------'   192.168.1.1/24   '--------------'
         |
         |
  VPN | 10.0.0.1/30
         |
    .--------------.
    | OPNsense2 |
    '----.----------'
         |
         |
  VPN2 | 10.0.1.1/30
         |
    .--------------.
    | OPNsense3 |
    '----.----------'
         |
         |
         .
WAN3 / Internet3

my goal would be to use the WAN3 for specific IP-Adresses from my LAN-Clients (on OPNSense1); So to NOT route them to the WAN on OPNsense1

I tried to go the easy way to define a new gateway in OPNsense1 that points on the OPNSense3 IP-Adress (where it should use the default routing) Since this is another subnet than directly attached i defined it as "far gateway" - at least that's how it's called in the GUI.
But it seems like there is "no way back" even though all OPNSense know how to reach every IP-Adress.

So my question was:

how do i have to configure the FWs to get the traffic the right way?
Config static routes on OPNSense 1, then 2 and include 3?
i somehow hoped that there is a nicer solution, so that i don't have to configure this routes on every system.
Especially since my routes are dynamic (due to the HA-Setup and multiple ways to a destination)

I hope this clears it up a bit?

if I got the traffic through WAN3 I can easily setup the missing routes for the additional VPN at OPNsense3

[defaultuserfoo]

Why don't you use OPNsense3 directly as gateway for LAN Clients after creating a VPN conntection between OPNsense3 and OPNsense1?  If you use openVPN, that already gives you a gateway, and you can simply use a firewall rule on the LAN interface to route the traffic through that gateway if it doesn't bother you that it would circumvent the routing table.  Or you could make the VPN gateway the default gateway.

Routes don't become suddenly dynamic when there are multiple routes to the same destination.  It may suffice to give routes (gateways) different weights, or perhapts you could use STP, or maybe OSPF.

What's the point anyway?  And have you considered using tor?

[wedge1001]

Of course I can add an additional routing-point from 1 to 3 (or in my case from 1 to 3 additional opnsenses), but why should i do this? this will make it even harder to figure out Problems in the Network.

to change the default gateway won't do.
especially since on every opnsense there is a local LAN attached. If I don't want to add lot's of manual routing this isn't feasable. Moreover I have to do this everytime I add another subnet (which actually happens quite often).

i'm propagating my routes via BGP; so of course the way from upper left to the lower right system can be different everytime something happens within the network (see the picture in the first post). Therefore routes do changes dynamically.

Why I want to do this: access geo-restricted sites without setting up a local VPN-connection on each needed opnsense system.

[defaultuserfoo]

Of course I can add an additional routing-point from 1 to 3 (or in my case from 1 to 3 additional opnsenses), but why should i do this? this will make it even harder to figure out Problems in the Network.

to change the default gateway won't do.
especially since on every opnsense there is a local LAN attached. If I don't want to add lot's of manual routing this isn't feasable. Moreover I have to do this everytime I add another subnet (which actually happens quite often).

i'm propagating my routes via BGP; so of course the way from upper left to the lower right system can be different everytime something happens within the network (see the picture in the first post). Therefore routes do changes dynamically.

Why I want to do this: access geo-restricted sites without setting up a local VPN-connection on each needed opnsense system.

The idea is to reduce the number of routing points to make things less complex.  Once you have solved the problem with the most simple case, it may be possible to apply the solution to more complex cases.

I haven't done anything with BGP or OSPF or any dynamic routing yet.  IIRC the difference between BGP and OSPF is that OSPF is for routing within the same institution/company while BGP is for routing anywhere over the internet, and thus it's more complex.

Do you have an OPNsense system at a place at which geo-restriction doesn't occur?  In your drawing it seems as if you want to use multiple VPN connections between all the OPNsense installations, i. e. on each needed OPNsense system, and you don't seem to want to use direct connections.

Perhaps a star configuration is better for what you're trying to do than a daisy-chained configuration.  At least it seems to make more sense, and it would give you less points of failure for each of the nodes because direct connections would be be used.

Anyway, I still don't understand why you would want to use a "far", i. e. unreachable (or whatever), gateway.  It would seem that you need to use reachable gateways to route your traffic through.  Or how do you expect the traffic to go over unreachable gateways?

I guess once you figured out how to route the traffic between two directly connected nodes, you may be able to change the routing dynamically.  And thinking of it, the routing must look different from the perspective of each node because each node has different neighbours, and to be able to use different routes, each node needs multiple connections to other nodes in the first place because otherwise, there would be only one route to take from each node, which isn't very dynamic.

I don't see that in your drawing.  Daisy-chaining nodes gives each node only one possible route to the endpoint at the end of the chain.