Hi everyone,
I am relatively new to OPNSense and just started few weeks ago to implement different network scenarios for testing, first in my closed home environment, and then on more complicated setups. I am facing some issues in understanding how to properly allow or not allow specific traffic between interfaces. In particular, in one of the setups I am working on at the moment, I have a Protectli Vault FW4c with 4x 2.5Gbit/s NICs, and I want to use 2 of those ports to attach Wifi6 EAPs and maximize the traffic towards the internet. As most of the traffic goes out I have managed to create VLANs on OPT1, OPT2 and LAN and bridge them so that a device connecting to either of the EAPs or to specific ports on the network LAN will be tagged to follow specific rules. Now as each VLAN runs its own subnetwork, and the physical devices are connected to different ones, I would like to allow the management of all network related infrastructure devices, only from IPs (a list of) coming from the LAN interface.
This includes the following:
1. LAN: 192.168.142.0/24 with OPNSense on .1 and switch on .2 (VLAN switch)
2. OPT1: 192.168.143.0/24 with OPNSense on .1 and EAP on .2 (an SSID for each of the VLANs)
3. OPT3: 192.168.144.0/24 with OPNSense on .1 and EAP on .2 (same as above)
While ICMP are enabled between all interfaces, I would like to allow TCP traffic towards the network_infrastructure (a Firewall Alias with all of the 6 above IPs) only from the LAN net addresses. I have setup a simple rule to do so, and to my surprise the log shows that the traffic is allowed out of LAN towards OPT1 (this is an IN rule) but then when it reaches OPT1 the default OUT rule "let out anything from firewall" kicks in, and instead of allowing to connect to the local address (i.e.: 192.168.143.2 on port 443) is actually forwarding the request to the WAN port...
The rule I setup on LAN is the following:
IPv4 TCP LAN net * network_infrastructure 22 (SSH) * * Allow to connect to infrastructure devices via SSH
And network_infrastructure contains the following IPs: 192.168.142.1, 192.168.142.2, 192.168.143.1, 192.168.143.2, 192.168.144.1, 192.168.144.2
I have been trying to understand why OPT1 would think that a request incoming from LAN net towards an OPT1 net address would be considered an OUT traffic, but I have found no valid answer so far.
Any help would be greatly appreciated, thanks in advance
only use OUT rules if you know what you are doing. normally IN rules work well.
Quote from: chemlud on July 25, 2023, 03:25:10 PM
only use OUT rules if you know what you are doing. normally IN rules work well.
Thanks for the quick reply @chemlud I guess my message wasn't clear 😉 I am not trying to use OUT rules, it is the default created rule "let out anything from firewall itself" which kicks in, even if in my opinion shouldn't because the traffic is incoming into OPT1 from LAN and as the connection is initiated in the LAN interface, it should continue with the TCP handshake and provide a connection. Instead what I can see in the live view log is the following:
Interface Time Source Destination Proto Label
OPT1 2023-07-25T15:48:26 192.168.142.100:52745 192.168.143.2:443 tcp let out anything from firewall host itself
LAN 2023-07-25T15:48:26 192.168.142.100:52745 192.168.143.2:443 tcp Allow to connect to infrastructure devices via HTTPS
My understanding is that the rule on LAN is kicking in as expected, but then OPT1 instead of replying as expected seems to consider the traffic OUTgoing... which doesn't make sense as the address is on the same subnetwork of OPT1. If I check in the diagnostic this is what I get:
Proto
Source
Gateway
Destination
State
Age (sec)
Expires (sec)
Pkts
Bytes
Rule
tcp 192.168.142.100:52769 192.168.143.2:443 SYN_SENT:CLOSED 9 28 7.00 448.00 let out anything from firewall host itself
tcp 192.168.142.100:52769 192.168.143.2:443 CLOSED:SYN_SENT 9 28 7.00 448.00 Allow to connect to infrastructure devices via HTTPS
udp 192.168.143.2:42462 255.255.255.255:29810 NO_TRAFFIC:SINGLE 194 29 20.00 16.66 K Allow all traffic towar
Which is strange as I wouldn't expect the SYN_SENT:CLOSED to happen...
Does this make sense to you?
But traffic towards hosts on the OPT1 network is outgoing through the OPT1 interface. In and out are always in relation to the interface in question.
Quote from: Patrick M. Hausen on July 25, 2023, 04:06:57 PM
But traffic towards hosts on the OPT1 network is outgoing through the OPT1 interface. In and out are always in relation to the interface in question.
Hey Patrick, thanks for your reply... ok I thought naively that OUT means out of the Firewall and IN means into the firewall, while it actually means IN and OUT of each interface. Then if that is the case the problem remains as the "let out anything from the firewall itself" is actually a very unrestrictive rule and it is higher in priority than the one I am going to write. Unless the "last match" vs "fist match" makes any difference, which it doesn't seem to at the moment.
If I want to restrict access to OPT1 only from LAN, but I need to allow an OUT rule from OPT1 towards LAN to allow the TCP connection to be established, wouldn't that expose the LAN (administrative network) to more attacks from OPT1?
I think I am more confused that I already was, thanks for opening more doors :-)
OPNsense is stateful. If you have a rule on LAN:
Source: LAN net
Source port: *
Destination: OPT1 net
Destination port: HTTPS, SSH, whatever, or *
Action: allow
That is all that is necessary to establish a TCP connection. The first packet from LAN to OPT1 will setup a state that will permit bi-directional flow of packets belonging to the same TCP connection but nothing else.
I'd try a packet trace on the target system before blaming the firewall in this particular case.
these "discussions" start here every other day or so. can we please have a topic pinned to the top "what is IN and OUT in firewall rules? and why does it matter"... just asking...
@chemlud What puzzles me most is why people are so confused about this - this is how firewalls have worked since day 1. Livingston, Cisco IOS, Cisco PIX (before ASA), Checkpoint, Sidewinder, FreeBSD, Linux, ... all of them.
Are there consumer devices that treat this different? Where do people get these ideas?
Quote from: Patrick M. Hausen on July 25, 2023, 04:16:45 PM
OPNsense is stateful. If you have a rule on LAN:
Source: LAN net
Source port: *
Destination: OPT1 net
Destination port: HTTPS, SSH, whatever, or *
Action: allow
That is all that is necessary to establish a TCP connection. The first packet from LAN to OPT1 will setup a state that will permit bi-directional flow of packets belonging to the same TCP connection but nothing else.
That is exactly what is in place, and the result that I am seeing is that the connection is initiated from the LAN address, but there is no return packet sent back from the OPT1 address. As shown in the log that I have posted above.
I didn't blame the firewall, I genuinely am trying to learn how to use it. I still don't understand why if the traffic is sent towards the address correctly (as I can ping the address, it connects to HTTPS if I plug in the network cable directly, and it sends a SYN request) and as you say that is all I need to do, then why there isn't a SYN_ACK sent back, what could the problem be?
Thanks
I have tried also to define 2 different floating rules, with direction: any for both HTTPS and SSH ports, on the specific network_infrastructure IPs, with the same results. I have tried to connect from the LAN address to the OPNSense instance on the OPT1 network, and that works, but it triggers the no-lockout-rule after the one that I have created to allow traffic with source LAN net towards one of the defined IPs.
So it would appear that there is something else missing, or the second rules shouldn't be triggered, right?
tcpdump on LAN, tcpdump on OPT1, tcpdump on both end devices ...
BTW what do you mean by "OPT1 address"? You are supposed to initiate the connection with the IP address of the device connected to OPT1, not the address of the OPT1 interface of your OPNsense ...
EDIT: just a shot in the dark - the "anti-lockout" rule does weird things with NAT for ports 443 and 22. I always disable it and simply make sure UI and SSH access is allowed where I want it. Try to disable that.
Firewall > Settings > Advanced > Disable anti-lockout
Quote from: Patrick M. Hausen on July 25, 2023, 04:54:14 PM
tcpdump on LAN, tcpdump on OPT1, tcpdump on both end devices ...
BTW what do you mean by "OPT1 address"? You are supposed to initiate the connection with the IP address of the device connected to OPT1, not the address of the OPT1 interface of your OPNsense ...
Yes, thanks for asking... I initiate the connection from 192.168.142.100 → 192.168.143.2:443 and it doesn't connect, it goes in timeout. I can't run tcpdump on the second one because it is a tp-link EAP and the address I am trying to reach is the admin interface of the device.
Even if I try using SSH, with a similar floating rule:
tcp 192.168.142.100:53274 192.168.143.2:22 SYN_SENT:CLOSED 7 28 6.00 384.00 let out anything from firewall host itself
tcp 192.168.142.100:53274 192.168.143.2:22 CLOSED:SYN_SENT 7 28 6.00 384.00 Allow SSH traffic from LAN to infrastructure devices
You can see I am connecting from an address in the LAN (192.168.142.0/24) to an address in the OPT1 network (192.168.143.0/24) and I get a SYN_SENT:CLOSED back...
Any other idea to try troubleshooting this?
Disable anti-lockout for a test ...
Quote from: Patrick M. Hausen on July 25, 2023, 05:07:41 PM
Disable anti-lockout for a test ...
Ok, I didn't lock myself out :-) And the connection to the IP address of the OPNsense on the network of the OPT1 interface works as expected. Therefore I guess it isn't a problem with the rule configuration but something else specific to the TP-Link management interface. To check that everything works there, I have connected directly to the AP using an ethernet cable, and I was able to login and verify that there isn't any management restriction related to the access via SSH or HTTPS. And still won't connect, any other idea?
Does the TP-Link have a proper default gateway? I.e. the address of your OPNsense in the OPT1 network?
You could try to configure outbound NAT on OPT1 with the source limited to LAN net - that will make the connection appear local to the TP-Link device.
Quote from: Patrick M. Hausen on July 25, 2023, 06:12:41 PM
Does the TP-Link have a proper default gateway? I.e. the address of your OPNsense in the OPT1 network?
Yes it has that configured manually as well as the IP address. I have set it manually to be sure it was correct.
Quote from: Patrick M. Hausen on July 25, 2023, 06:12:41 PM
You could try to configure outbound NAT on OPT1 with the source limited to LAN net - that will make the connection appear local to the TP-Link device.
I will try this, but so far it appears there is no restriction on the access via network, even though they suggest to use the Omada Controller interface, I have decided not to, as it will add more unnecessary complexity to the network management.
Thanks for your help :-)
I am back to my original point now, after having experimented with multiple different setups. I will try to summarize again what is my intention, and what I am trying to achieve. I will also share what I have already tried, and I am looking forward to any suggestion that would come from the group:
IntentI want to setup a network with different VLANs to isolate traffic, and I want to allow the management of the network infrastructure only to admins (only to a specific set of IPs → Firewall Alias: admin_ips).
Current SituationI have setup the following VLANs which are successfully connecting to the internet and do not have access to each other:
- UNET: user network, going from access points to port OPT1 and OPT2 directly on the firewall/router, as well as specifically tagged ports on the main switch connected to the firewall/router LAN port. Traffic from the UNET to the internet works, and I have also enabled traffic within the UNET so that devices can connect to each other and use different services. I will restrict this further in the future, once the rest works.
- GUEST: guests network, only Wifi and only towards the internet.
- SECD: secure devices such as cameras, motion detectors, doors and windows sensors, isolated only towards the internet.
- MEDIA: all smart TV and players, which are accessible from the UNET only.
- HAUTO: home automation and IoT all small quirky and dangerous things, which are also accessible only from UNET and can only go to the internet.
I didn't want to setup a dedicated management VLAN because it is quite overwhelming for only a couple of IPs to manage, and because I have the access points attached to the firewall directly I had to setup VLAN bridges between OPT1, OPT2 and LAN specific VLANs, and assign different subnetworks with DHCPs (both 4 and 6 in some networks).
Open ProblemI am still not able to access from a specific IPs set (admin_ips alias) network infrastructure devices, from different subnetworks. If the devices are in the same subnetwork of the admin_ip(s) then it is reachable, otherwise it isn't. The behavior is pretty much what I have captured below which shows the "live view" first and then the "diagnostic view" after:
Quote from: tumma72 on July 25, 2023, 03:52:13 PM
LIVE VIEW:
Interface Time Source Destination Proto Label
OPT1 2023-07-25T15:48:26 192.168.142.100:52745 192.168.143.2:443 tcp let out anything from firewall host itself
LAN 2023-07-25T15:48:26 192.168.142.100:52745 192.168.143.2:443 tcp Allow to connect to infrastructure devices via HTTPS
DIAGNOSTIC VIEW:
tcp 192.168.142.100:52769 192.168.143.2:443 SYN_SENT:CLOSED 9 28 7.00 448.00 let out anything from firewall host itself
tcp 192.168.142.100:52769 192.168.143.2:443 CLOSED:SYN_SENT 9 28 7.00 448.00 Allow to connect to infrastructure devices via HTTPS
Which is strange as I wouldn't expect the SYN_SENT:CLOSED to happen... what I have tried so far:
- Created floating rules for HTTPS and SSH with source admin_ips to allow incoming on both protocols (which is the "allow to connect from infrastructure devices via HTTPS" above.
- Created specific interface/IP rules to allow IN connection from the admin_ips with the same result as above.
At this point I am a bit puzzled, as the log shows that the rule is triggered and the traffic reaches the second subnetwork (VLAN) but then doesn't come back. I have checked that all network have configured - both via DHCP and on the devices themselves - the right gateway address as their own network IP address of the OPNsense firewall. There aren't any competing rules in place, I have actually removed everything that was there to test this specific behavior, and I am still not getting anywhere.
This is what doesn't work:
a. I am connected with an admin_ip in the 192.168.148.0/24 network, and I am trying to connect to a switch which is on 192.168.148.0/24 network both on port HTTPS and SSH, but the traffic doesn't go through. I am able to ping the device though, which means that ICMP traffic is going through, and I have a rule for that which states that all ICMP requests from admin_ips are allowed in, and then the allow everything out of the firewall allows for the response to come back.
b. I am not sure how to configure the FailoverWAN properly, meaning that I have followed the guide, and I have a working FailoverWAN with a fiber connection and a backup LTE connection, that works, but I had to add a rule to all subnetworks to use the FailoverWAN (which is a gateway group) to access the internet. Without that rule the connection towards the internet won't work.
Any help would be greatly appreciated :-) Thanks in advance!
I suppose I am writing to myself for the next one having similar issues...
But as I make progress I am adding information in the hope that will be helpful.
Quote
a. I am connected with an admin_ip in the 192.168.148.0/24 network, and I am trying to connect to a switch which is on 192.168.148.0/24 network both on port HTTPS and SSH, but the traffic doesn't go through. I am able to ping the device though, which means that ICMP traffic is going through, and I have a rule for that which states that all ICMP requests from admin_ips are allowed in, and then the allow everything out of the firewall allows for the response to come back.
SolutionAll TP-Link smart devices have an explicit setting for the route (other than the gateway) and have an explicit option for L3 connection which basically means they are routing to another subnetwork and allow devices coming from different subnetworks to connect. I have enabled the L3 options on both Access Points (EAP670) and the switch (TL-SG2008P) and I am now able to access from other networks. The reason why I didn't see this before is because it isn't in the Management or Security part of the interface but rather hidden on 3rd level menus...
Anyhow I had to add a VLAN ID: 1 to be able to manage all devices, because they are configured in such a way that you can't remove it, but I found a rather secure way to enter the management VLAN using a tagged port on the switch, that by default assigns the user network VLAN tag, while at the same time allows the already tagged VLAN 1 traffic to pass through. To create the VLAN for management spanning on all networks I had to create one VLAN connection with the tag 1 for each of the physical interfaces to which there are infrastructure devices connected and then I had to create a VLAN Bridge interface to include them all, and finally enable DHCP on that bridge. It seems to be a bit cumbersome but it works so far. I prefer to have DHCP running and then assign static leases rather than having static IPs, it makes the management easier and I do not have to access all of the devices individually to change IPs in case it is necessary.
Now the next challenge... making Apple devices and HomeKit work while being on different subnetworks... I will check around if there is already documentation about this, or I will start another post.
Thanks to everyone who contributed to help out...