Connection Problems in LAN

[RatherOldMan]

Hi All.

I have these two LANs: MAIN and SUB.

Because clients from MAIN have to access some services in SUB and the other way around the SUB-Firewall is connected directly to the MAIN LAN with an interface.

Now I want to exchange the MAIN Firewall because it is an VERY old pfSense 2.2.5.
So I configured two new OPNsense HA Firewalls.

Now I have the problem:
On a client in the MAIN LAN i try to start a application from a share in the SUB LAN.
I can access the share - the connection speed is good. I can transfer files etc.
But the application starts VERY slow and is unusable.
The application self trys to connect to a service on the same host where the share is located.
It feels like the connection is stuttering.

If i use the OLD MAIN Firewall (pfSense) I do not have that problem.

SUB Firewall is OPNsense 20.1.7
NEW MAIN Firewall is OPNsense 23.1.1_2 HA

I added the Gateway 10.0.1.12
I created a static route to the SUB LAN over that Gateway

Firewall > Settings > Advanced:
Disable force gateway: [X] Disable automatic rules which force local services to use the assigned interface gateway
Static route filtering: [X] Bypass firewall rules for traffic on the same interface

In my (outdated?) simple knowledge from what is going on:

- the client asks his default gateway (MAIN) where he will find the SUB-Host
- the MAIN-Firewall tells the client that he should go to the SUB-Firewall
- the client connects to the SUB-Firewall and can access the SUB-Host

I tested that routing-idea on the client and create a route on the client to the SUB LAN over the SUB Firewall.
The application starts very well, looks good!
But why is the Gateway (MAIN) not answering THIS question about the route?

I took a deeper look at the differences between the NEW and the OLD and find
net.inet.ip.redirect
is ACTIVATED on the OLD - and DEACTIVATED on the NEW.

So I thought: yes, that could be.
The client receive an ICMP Redirect from his gateway (MAIN) so that he tries to connect over the SUB-Firewall.
I activated net.inet.ip.redirect = 1
But nothing changed.

BTW: Its no difference if i use the CARP IP or the IP from the Firewall itself.

Its still stuttering.

So: BUG? Or no ICMP Redirects anymore because of security reasons? Or do I have a total misunderstanding from that scenario?

Hints are very welcome

Thx

[Patrick M. Hausen]

Default to off is a good one, because people have been trying to play tricks on devices with ICMP redirects. I enable them on trusted networks like LAN because they make routing more efficient.

[RatherOldMan]

Hi,

I agree - Default Setting to OFF is a good idea. A lot of users won't need it.

But can I somehow test if the NEW MAIN-Firewall really sends a ICMP Redirect? So that I have to continue my search for the problem?

Atm I don't know if it send a ICMP Redirect.
On the search in the internet I read something about ICMP Redirects not working in FreeBSD. But that was about Version 11 if I remember correctly. Currently I don't find that source again.

So I try to ask more detailed:
Is there a log where I should see that ICMP Redirects the MAIN-Firewall is sending?
Is there a BUG so that ICMP Redirect currently is not working?
Or is there a decision not to use ICMP Redirects anymore because of security reasons so that it couldn't be enabled?
Or do I have a total misunderstanding from that scenario because my skill of knowledge isn't high enough?

Wave,
The RatherOldMan

[Patrick M. Hausen]

tcpdump will show you the packets sent. I am not aware of any open bug.

[RatherOldMan]

Thx pmhausen.

The client receives ICMP Redirects. So the problem must be something else.

I guess its not the problem to start the application itself from the share.

I guess it has something to do with the Database-Connection, the application wants to connect to a Database-Server in the SUB network. Because it didn't receive a (good) result - it can't start well.

Another Software which work with Client-to-Database-Server looses connection to the server in the SUB network if I try to request more data at startup. But only at the first start of that client - further on it work.

Wave,
The RatherOldMan

[RatherOldMan]

Isn't this asymmetric routing?

Client in MAIN
> Firewall MAIN
> Firewall SUB
> Server in SUB

Server in SUB
> Firewall SUB
> Client in MAIN

Is there a better way to connect two local networks where both uses different OPNsense as Default Gateway?

[Patrick M. Hausen]

That will not work due to the fact that OPNsense is a stateful firewall. If the client in one network initiates the connection to a server in another one through firewall A, the return packets must go through firewall A, too.
So you must ensure by network design that routing is symmetrical.

[RatherOldMan]

Thx pmhausen,

can i solve that issue with ICMP Redirects?

The Clients gets the Information from A to directly connect to Router B.
So that the resulting connection is symmetric:
Client > B > Server
Server > B > Client

You said that you enable ICMP Redirects in trusted networks.
Maybe I misconfigured the OPNsense to send ICMP Redirects?

net.inet.ip.redirect = 1

Is it more than this?

Can someone pls confirm that ICMP Redirects are working with OPNsense 23.1.3_4?

Wave,
The RatherOldMan

[RatherOldMan]

Sorry I'm a bit desperated.

If I have a problem I couldn't really understand I'm in heavy doubt about my technical understanding at all.

I tried this:
https://docs.netgate.com/pfsense/en/latest/routing/static.html#asymmetric-routing

So i create such rules on the LAN Interface and on Floating: ALL TCP Flags and Sloppy state.

And - it worked. I will test it further tomorrow - now I have to make the fodder for our two dogs.

Wave,
The RatherOldMan