Connections are disappearing in firewall

[LoC]

I got a few virtual machines. All these machines are running in their own VLAN. Those VLANs should all be separated from each other, so I installed a floating rule which blocks all requests in direction "out" with the destination <alias for all VMs>. I also allow all VMs to establish connections in the direction "in" to any destination.

I also got a rule that allows ICMP in the direction "out" with the destination <alias for VMs> and a rule that allows SSH in the direction "out" with the destination <alias for VMs>. Those should be all releavnt floating rules, I also got some interface specific rules for the different functionalities of the VMs.

My Problem: this setup works with 4 out of 5 virtual machines. I can ping them, I can connect to them via SSH, but with one machine nothing works. interestingly,this machine also cant connect to the internet , although it should be able to do this. The other machines can connect to the internet.

I can ping this VM using the diagnostic tools in the firewall and vice versa. But all connections from my laptop only reach the opnsense. When I look into the "live view", i can see the connections in the direction "in" to the firewall, but no matching connections "out".

What am I doing wrong here? I look forward to any answer!

[Saarbremer]

Hi,

what you are doing wrong here? Well, how about providing more details about your setup and your VMs IP configuration.

Furthermore, try to find out why 1 out of 5 machines do not work as expected. This observation makes it very unlikely to have a systematic failure or "wrong" setup.

P.S.: Why floating, why out? I never understood why explicit info shall be buried behind the curtain and fall on your feet once you change something.

[LoC]

First of all, thank you for your answer.

My virtual machines are attached to a network interface on the hypervisor. The vlan tag is set on the hypervisor, the VMs then obtain their IP addresses via DHCP. The network configuration of these machines and their interfaces on the hypervisor is excatly the same. Furthermore, all VLANs and their assigned interfaces on the OPNsense are configured the same way with a static IPv4 address for the OPNsense interfaces and no IPv6.

try to find out why 1 out of 5 machines do not work as expected.

Well, thats exactly my problem: i have no idea. I tripple checked (at least) that the interfaces on the OPNsense are configured the same way and that the interfaces on the hypervisor are set up equally. The VMs have the same network configuration anyway - they are set up to just get their address through DHCP, nothing else.

Why floating, why out

I thought that this is the best way to do it? The configuration of "denying all connections to the VMs, allowing the VMs internet access, allowing SSH to the VMs" is supposed to be the same accross all the VMs and their VLANs/interfaces, so I thought that floating rules would make sense. about the "why out": because I thought I wanted to deny access "out" from the opnsense to the VMs. Maybe there is a better way to do this, I would be happy to learn about it.

[Saarbremer]

Hi,

finding out why the setup is not working as expected would mean:
* Switch on logging for all affected rules (pass or reject/block, no matter)
* Go to rules -> live view during a successful test and a failing test (with your one special vm)

Can you spot any differences? If yes what are they?


Regarding out and floating: As long as it works, feel free to go with it. But be aware of nasty side effects once you want to change something. E.g.: What if you creaete a new management VLAN that is supposed to run periodic health checks on your VMs on port 443?

You'd need to update your OUT rule and another IN rule on the new VLAN. But again, live view would tell you that. As described above. So please have a look into your system.

[LoC]

I already enabled logging for every rule I could find. When I try to ping the "not-working" VM and look into my live view, i can see the ping in the direction "in" to the firewall, but no matching ping "out". When I try to ping one of the "working" VMs, I can see my connection "in" as well as my connection "out".

[Seimus]

In regards of best practices of rules IN vs OUT.

In enterprise as well we prefer to do IN (inbound based rules), this is just common practice. You either filter the traffic on the inbound direction of a interface or a ZONE. In case the rule is permit you let it usually out without any other filtration from perspective of the FW if your main achievement is to block or permit.

OUT rules (outbound based rules) are used usually only for very specific cases.

Now why to use IN and not OUT, well simple answer is to prevent any other processing. If your ideology is let anything IN on the ingress interface than restrict it on OUT on the EGRESS interface, the packet will go thru several processing steps.

You can check this thread created by Monviech, which he did an amazing job to map processing of the packet thru OPNsense. To see differences when you block IN or OUT.
https://forum.opnsense.org/index.php?topic=36326.0

If you wanted to apply similar rules to multiple interfaces you could use as well the Grouping feature in OPNsense, which is fantastic. The rule application order is per OPNsense documentation
System > Floating > Group > Interface

I for example did bundle all my VLANs except management to 1 common group and defined base rules such as DNS, HTTPs etc. to them. And rest is fine tuned per interface. Works like a magic and you dont need to have XYZ duplicate rules per interface.

I already enabled logging for every rule I could find. When I try to ping the "not-working" VM and look into my live view, i can see the ping in the direction "in" to the firewall, but no matching ping "out". When I try to ping one of the "working" VMs, I can see my connection "in" as well as my connection "out".

You mean by this that you don't see any out rule for that non working VM? Or its hitting a deny?

Regards,
S.

[LoC]

Thank you very much for your elaborate answer. I think I will adjust my configuration according to your recommendations in the future!

You mean by this that you don't see any out rule for that non working VM? Or its hitting a deny?

I dont see any out rule for that VM. With the working VMs, I see the "allow-icmp" out rule that I set up (just like expected). I dont see any "block/reject" rule either, unfortunately..

I just noticed, I also got a rule that allows TCP traffic to the HTTPS port of the non-working machine. For that rule, I actually see the "in" rule as well as the "out" rule in the live view. It says that it lets this traffic pass. but when trying to establish connections to that port (calling the webserver that is listening there), the connection times out, which is even more confusing for me..

[LoC]

I am even more confused right now. As I wasnt able to find a solution, I basically deleted the whole setup with the floating rules and everything. I then proceeded to create a group with no "out" rules and some "in" rules allowing traffic to dns, ntp, that kind of stuff. But the result is still the same: I just cant ping that machine. Nothing changed, even though I reconfigured everything I thought to be concerning this problem.

I can still ping the other machines though.

[FullyBorked]

Here is what I would do. 

First a way to understand opnsense firewall is everything is relative to the firewall itself.  Once you understand this firewalling is simpler here.  Everything is by default allowed out from the firewall.  So you just need to allow traffic in to the firewall. 

I'd just remove all your floating rules and any rules you've created on those new vlan interfaces.  If you want to ping, simply create an inbound rule on the interface you are pinging from, source any, destination vlan net you're pinging to allow ICMP.  And test your ping.  That should help you learn something.  If it's allowed in the firewall log, then it's blocked at the VM itself or at the hypervisor. 

Attaching an example of what this rule might look like from my server vlan to my DMZ network. 

[LoC]

Thank you for the suggestion. But unfortunately that is what I already did: I deleted all the floating rules; and there was a rule which allowed me to ping the VMs (yes, also the trouble VM) enabled all the time. The problem is that the ping gets into the firewall (I can see that in live view), but it does not get out. It isnt denied or anything, the "out" part of the ICMP request (is it called a request?) just does not appear in the live view. I know that this could mean that logging for the rule blocking the "out" part just is not enabled, but I checked everywhere and there is no rule that rejects anything.

EDIT: in the meantime, I also deleted the VM and set it up again. It now runs with a live image. I also deleted the interface (on the opnsense as well as on the hypervisor), tried with different VLAN tags and with a different set of IP addresses to be used on the interface. None of this helped in any way, the result was always the same..

[FullyBorked]

It has to be blocked somewhere, slow down and only check one thing at a time. Making multiple changes at once will only increase your frustration and confusion 

You can also create an ICMP rule with source and destination to ANY on the source interface/vlan.  Make sure to enable logging as suggested above.  Send your ping, if you still aren't seeing it in the logs, you may have a configuration issue somewhere else. 

Edit: with the above any any ping rule in place, do some other checking and trace route tests.  Can you ping the firewall itself?  Can you ping www.google.com?  Can you ping 1.1.1.1?  Where does a trace route fail to the VM you are trying to ping? 

[FullyBorked]

Thank you for the suggestion. But unfortunately that is what I already did: I deleted all the floating rules; and there was a rule which allowed me to ping the VMs (yes, also the trouble VM) enabled all the time. The problem is that the ping gets into the firewall (I can see that in live view), but it does not get out. It isnt denied or anything, the "out" part of the ICMP request (is it called a request?) just does not appear in the live view. I know that this could mean that logging for the rule blocking the "out" part just is not enabled, but I checked everywhere and there is no rule that rejects anything.

EDIT: in the meantime, I also deleted the VM and set it up again. It now runs with a live image. I also deleted the interface (on the opnsense as well as on the hypervisor), tried with different VLAN tags and with a different set of IP addresses to be used on the interface. None of this helped in any way, the result was always the same..

Don't get too caught up in the out rules, for now at least don't create any "out" rules.  You should have one default rule "let out anything from firewall host itself " that's floating that will cover packets traversing the firewall to other networks. The default rule should have logging enabled as well, if not enable it.

Think of it this way [source vm] ----> (inbound ping rule) ----> [firewall] ----> [destination vm].  Will be all that's needed to allow this traffic. 

[LoC]

I added screenshots of the rules on the interface I am currently on and the group interface of the VMs. The interface of the trouble VM has no rules set at all. From my machine, I can ping the firewall, google, other VMs, everything except the vm causing problems. From the VM, I cant ping anything except the firewall. From the other VMs, I can ping everything just as well as from the machine I am using right now.

When trying to trace route, it fails at the IP address of the firewall on the interface my machine is on. I think this matches with my former observations in the live view?

[FullyBorked]

I added screenshots of the rules on the interface I am currently on and the group interface of the VMs. The interface of the trouble VM has no rules set at all. From my machine, I can ping the firewall, google, other VMs, everything except the vm causing problems. From the VM, I cant ping anything except the firewall. From the other VMs, I can ping everything just as well as from the machine I am using right now.

When trying to trace route, it fails at the IP address of the firewall on the interface my machine is on. I think this matches with my former observations in the live view?

I'm confused, if the interface the VM is on has no rules it shouldn't be able to ping anything.  Are you sure your vlan tags are being applied correctly?  Thinking there is a configuration issue with your vlan not putting the VM on the correct network.  Would explain why your rules seem ineffective. 

If you have group rules you should still be able to see them on each applied interface by expanding the hidden rules at the top.

[LoC]

I'm confused, if the interface the VM is on has no rules it shouldn't be able to ping anything.

thats right, but the group the vm is in is allowed to ping (second screenshot)

Thinking there is a configuration issue with your vlan not putting the VM on the correct network.  Would explain why your rules seem ineffective.

But it can ping the firewall and vice versa. And it obtains the ip address it is supposed to obtain via dhcp