[Solved] routing from one interface to another

[remd]

I'd like to route one physical interface to another and so far I haven't found a way to make it work.
My setup:
4 opnsense units (17.7.6), a 1st line of two in CARP then a 2nd line also in CARP. There is a DMZ switch behind the first line and a LAN switch behind the second line. The 2nd line is connected to the first line firewalls.  (on the fiber port IX1 to IX0)
There are a number of VLANs configured on the 1st and 2nd line firewalls and this is working fine.

I have however a guest wifi network going on another physical port from the 1st line of firewalls to the 2nd, on port igb3 on both firewalls, and this goes fine. The problem is that I then need to route the guest Wifi through the same fiber port (IX1) as the others to route it to the sole fiber cable going to the different floors, and I cannot get the traffic to go from the igb3 port to the IX1 vlan on the 2nd firewall.
If I ping from guest wifi IX1 VLAN outside or even to the igb3 port on the first firewall, and packet capture it, I can see the ping going to the first interface and coming back to the 2nd line of firewall on port igb3, but then it doesnt go to the vlan on IX1.
(I've setup a static route on the first firewall which routes the traffic back to the 2nd line for that network and that works fine)

So far I've setup a gateway with the IX1 vlan gateway and forced any traffic going to that network coming into the igb3 port on the second firewall to use that gateway, in the firewall rules for the igb3 interface, but it is apparently not doing it.
Does anyone know if this is the way to do this, or if it should be done differently ?

I also tried to setup a static route for that network in the system, routes section, but it wont let me as it mentions that there is already and interface configured for that network.

Does anyone have any advice on how to configure this ?

[remd]

...anyone?

[remd]

One thing that is strange is that another VLAN on IX1 can access the igb3 port.
The firewall rules seem the same and the logs don't show that it is blocked, it seems to be a routing issue....only from that one VLAN on IX1.
Will investigate a bit further, or maybe end up re-creating the VLAN

[remd]

ok the difference with the other vlan is that the return path from the first firewall is over IX0 not igb3.
It should still work, but that is the difference..

[remd]

ok  igb3 can see the ix1 vlan on the second firewall....but there is another issue now:
the first firewall can see igb3 on the second firewall...but not the vlan on IX1 on the second firewall, although there is a route configured for this, the firewall logs shows its going to igb3 on the first firewall which should then route it to igb3 on the second firewall (static route, which it can see), and then route it from igb3 on the second firewall and the ix1 vlan (which it can see now)

So the static route seems to be applied for the ix1 vlan as the request is sent to the right interface on the first firewall but then it is not routed to the igb3 interface on the second firewall (static route gateway which it can see)...

[remd]

ok so I realized something else, when a gateway is created it seems to be taken in consideration whether it is configured in an interface or a rule or not.
If I setup a gateway with the igb3 ip on that interface to force the ix1 vlan to go through it (which apparently is not needed), then igb3 cannot see the ix1 vlan anymore....even-though I haven't configured igb3 to use that gateway !

so now igb3 and the ix1 vlan see eachother on the second firewall but a request from the first firewall to the ix1 vlan will go to the igb3 interface on the first firewall and stop there (even with the static route to igb3 on the second firewall)

Another thing, I changed the ip range to see if it wasn't anything related to that range that was sticking anywhere, but that didnt help.
Then when I did that all the dhcp servers where in "recover-wait" state "partner-down" although I only changed one interface ??
And pinging from one interface to another is ok as always but all the dhcp servers ard down now.
I have tried to restart the service and reboot but that hasn't helped so far, hopefully they will come back, but not very reassuring..
UPDATE: the dhcp servers are back but it took them 15-20min to get back to normal

[Oxygen61]

Hi remd,

it's really hard to follow your explanation since there is no picture or anything.
It would help alot if you could create a picture of your network connections with interface names and so on.

Since i can't really help you by now i will just state some basics regarding VLANs and routing:

1. VLANs / physical interfaces and subnets created on the SAME Firewall don't need any static routes, since the Firewall is already directly connected to every one of them. Every interface is known to the firewall and it just routes the traffic between them based on firewall rules.

2. Whenever there is routing between subnets/VLANs NOT directly connected to the firewall, like routing to subnets of another firewall, you will need to create Firewall rules AND static routes pointing to that subnet/VLAN subnet with the "Out-gateway" used to reach the subnet.

3. whenever there are VLANs used and created you should really try to get rid of traffic, which is send through physical interfaces, since you are using VLANs and the physical interface is just the "Carrier" of every logical VLAN Interface ON that specific physical Interface. In the end you should even be able to delete the assignment of that physical Interface, since you are using VLANs. (Just make sure that there is one management VLAN for Web/SSH administration purposes.

4. Every route can be found here: System > routes > status
There you will see any "direct" connection, named link#X or your default route / Gateway of last resort.
If any "Destination Subnet" to any other Router/L3-Switch or Firewall can't be found there u can try to check if you will need to create a static route first to reach the subnet.

I probably didn't help you at all, but i really wanted to state some basics.
Please make a picture of your setup so that we can analyse the problem in more detail.
Don't forget to write down interface names and write down which interface is the carrier of which VLAN.

Best regards,
Oxy