[SOLVED] Help needed in setting up opnsense with multiple ETH ports

[jw64]

Hi,

I am running an opnsense machine with 6 ports. My setup is as the following:
    ETH0: WAN
    ETH1: LAN, 192.168.1.x/24
    ETH2: LAN, 192.168.2.x/24
    ETH3: LAN, 192.168.3.x/24
    ETH4: LAN, 192.168.4.x/24, this is for IoT devices
    ETH5: LAN, 192.168.5.x/24, this is for guest access
Each of the LAN subnets ETH1-ETH5 has its own gateway (192.168.x.1) and corresponding DHCP range.

I want ETH4 and ETH5 can access internet but not other subsets. I set up these two as guest nets using instructions in https://docs.opnsense.org/manual/how-tos/guestnet.html. These worked as expected.

I want ETH1/ETH2/ETH3 can access internet and all other subnets, so I created firewall rules to allow any IP traffic from these subnets to any destination. While traffic can reach internet, it doesn't work to reach nodes in other subnets, e.g., a node in ETH2 cannot reach any node in ETH1. How to change setup to fix this problem? I know I can create a bridge for ETH1/ETH2/ETH3 but I read that bridging in opnsense is not efficient and not suitable for heavy traffics so I'd like to get away from bridge if possible.

Thanks.

[netnut]

While traffic can reach internet, it doesn't work to reach nodes in other subnets, e.g., a node in ETH2 cannot reach any node in ETH1.

Start by posting the _actual_ (detailed) rules you configured for both eth1 and eth2. There must be a config error, if you fix that you probably fix all other issues too (might be same mistake).

[jw64]

@netnut. Thanks for your reply. Please see my firewall rules in the attached images and let me know what other configuration information should be examed.

[jw64]

File wall rules for ETH4 (ETH5 is similar). ETH1 has one additional anti-lockout rule in the automatically generated rules.

[netnut]

Your rules look pretty straight forward and allowing everything, so that's probably not your issue. The networks are directly connected to your OPNsense box, so routing shouldn't be a problem.

The only thing I can think of at this stage: ETH2 is labeled as WiFi, which assumes a wireless network. Is there some Access Point in this network ? How is that connected to OPNsense ?

How do you test

...a node in ETH2 cannot reach any node in ETH1

[jw64]

I think I found the issue. When I was testing if I can reach the ETH1 subnet from ETH2, I moved the network cable from ETH1 to ETH2, and tried to reach the web portal 192.168.1.1. That doesn't work.  However, if I connect another machine to ETH1, then I can reach 192.168.1.1 from ETH2 as expected.

So it looks that I may not have problem in the network setup. The issue is that I could not reach web portal 192.168.1.1 when ETH1 is not connected to an active machine. Is this normal? Is there any configuration change that can make the web portal 192.168.1.1 alway available?

[netnut]

The issue is that I could not reach web portal 192.168.1.1 when ETH1 is not connected to an active machine. Is this normal?

Yes, are you aware of the OSI model ? Your patch cable into OPNsense ETH1 is layer 1 (physical connection) and triggers the so called "CARRIER". If there's no signal (no patch cable in the port) all other layers, MAC (L2) and most importantly IP (L3) won't be active.

If I understand your problem right you want ETH1 to be always on, with or without an (active) patch attached (in other words UP without CARRIER). To be honest I don't know if that's a configuration option in OPNsense, so maybe another forum member can answer this...

Another approach would be running OPNsense management on a loopback address, so it will be always on and with the right firewall rules only reachable for the segments/networks you want.

[Patrick M. Hausen]

The issue is that I could not reach web portal 192.168.1.1 when ETH1 is not connected to an active machine. Is this normal?

Yes, it is. If you unplug an interface it goes "down" and its IP address is not reachable from anywhere else. Perfectly normal behaviour of routers - which OPNsense is in the end.

Normally firewalls are connected to infrastructure systems like switches that "never" go down. So this is not a problem. Why are you using a firewall port for a single PC?

There is no real configuration solution on OPNsense. The "infrastructure" mention above hints at a solution. My OPNsense is connected with two ports configured as a LAGG interface to my single central switch. All interfaces but WAN are VLANs on top of that.

if that sounds way too complex for you, you might think about a "smart" but not fully managed switch that can do port based VLANs and connect all your OPNsense interfaces to that.

We can surely give better suited hints if you describe your ultimate goal in some more detail. You now learned that a dedicated firewall port for a single PC is not the best solution

P.S. Much of this network and router ws. switch mumbo jumbo is in fact historical. 20 years ago router ports were 10 times as expensive as switch ports. With the advent of "layer 3 switches" and other combined architectures the landscape changed completely. So back in these days nobody would have even thought of "wasting" a router port for a single connection.

[jw64]

Thank for the information. I was using single PC for configuring and testing the firewall box.
I think my problems are solved now.