Routing issue between interfaces on opnsense firewall

[firewall_newbie]

Hi

I have an issue where I see traffic come in on one interface of the firewall but then it does not go out on the another interface. I have permit rule for the traffic but still I do not see traffic on destination interface of the firewall. Any help will be much appreciated. Below is scenario:

client---->if1(opnsense)------>if2(opnsense)----server

Here client traffic is seen on if1 of the opnsense firewall and server is on same network as if2 of the opnsense. There are no drop logs and I have a permit rule so it is not permissions issue but for some reason I do not see traffic on if2 of the opnsense.

I did packet captures and I see packets on if1 but not on if2.

Thanks

[meyergru]

Maybe you had if1 first as the preconfigured WAN interface and the default NAT rules are still in place? NAT rules are prioritzed higher than normal firewall rules.

[firewall_newbie]

I am not sure what WAN interface means but this is sub-interface for a VLAN and no NAT rules were configured. This interface was configured to connect to another LAN segment. This interface has gateway attached to it. So basically I try to connect from a client to a server behind this firewall.

client traffic enters firewall on if1(with gateway attached to it) and destination is the server which is directly connected to if2. I see client traffic on if1 but when I capture packets on if2 I see no traffic. At this point I am only concerned about traffic from client to server.

[meyergru]

When you first install OpnSense, it has a default setup for two interfaces, one for WAN with NAT rules and one for LAN. There are some default rules in place to make life easier for beginners, like an "allow any" rule for the first LAN interface.

But never mind, just inspect Firewall: NAT: Outbound. If there are any automatic or manual rules for your if1, you have identified the culprit, because you cannot make a connection against an outbound NAT rule (other than creating a DNAT rule).

[firewall_newbie]

sure will inspect the NAT rules. But I am not clear if outbound NAT rules will impact traffic inbound on same interface. The connection is initiated from a client that is received by firewall on if1(inbound) and target is a server directly connected to if2. So if there are any outbound NAT rules on if1 that should not match traffic inbound on if1.

[meyergru]

Correct so far. But all response packets for your inbound traffic will go through NAT. So even if you allow inbound traffic, answers will not come from the server, but from the if1 interface address. Therefore, that still does not work. You would need DNAT rules for that.

The way you set this up, you obviously want to use OpnSense as a pure router between two arbitrary subnets, yet you may have NAT in place, which is a hack to connect non-routeable networks with the global internet. That won't work (if my guess is right).

[firewall_newbie]

Correct I would like opnsense to behave like a router with no source or destination translation. I checked there are outbound NAT rules but for interface which is my gateway to internet and not if1. But again:

If with outbound NAT on an interface the traffic received will be source NAT'd but destination would remain same. So in my case destination is server on if2(same network) so I should still see packets on if2(packet capture) with source as NAT'd IP on if1 and destination as target server.

[EricPerl]

The interface with the GW to the internet is typically WAN (by default).
Also by default, the 2nd interface is LAN. Then OPT1 and so on.

Assuming both systems have IP on interfaces with distinct subnets, logging enabled on all rules, you should see:
IN traffic on IFL1, OUT traffic on IFL2 for the request.
The IN on IFL2 followed by OUT on IFL1 for the replies.
The FW live view should show you the request part. You have to do packet captures to see the replies.

When the interfaces are set up properly, you typically don't have to do anything to get routing to work.
You might want to share that part.

[firewall_newbie]

OK here is what I see:

With packet capture on if1 and if2, I see client to server requests on if1 but not on if2. In live view I see traffic IN on if1 but no OUT on if2 so it is sure packet arrived and was logged by if1 but thereafter it got lost somewhere between if1 to if2 within the firewall itself. If it helps I can snip configurations on if1, if2, NAT, firewall rule for interesting traffic here. I can also snip packet captures.

[dseven]

It seems likely that you don't actually have a firewall rule to allow the connection that you're attempting. Could you show the rule that you think should allow it (and state which interface the rule is on)?

[firewall_newbie]

I am not able to paste the rule on interface if1. Is there way to paste ?

[dseven]

You should be able to attach an image (screenshot).

[firewall_newbie]

Please find attached firewall rule on interface that receives the traffic from client

[dseven]

Reading back through the discussion, you said something that made me pause - "So in my case destination is server on if2(same network)". What do you mean by "same network"? You can't have two interfaces on the same network (and expect things to work)...

[firewall_newbie]

The target server is on same network as if2. In a sense server is directly connected to if2(same network). Why won't this work ? It is similar to having two hosts on same network so in this case one is server and other is the interface if2 on firewall. This makes them both part of the same broadcast domain and thus need no explicit routing to talk to each other.
Now when traffic from client enters firewall on if1 packet would be like:

src: 10.28.140.50
destination: 10.10.30.13
if1: 10.28.140.49
if2: 10.10.30.2

In this case when firewall sees destination to be 10.10.30.13 it knows it needs to send it to if2 and then ARP broadcast on if2- looking for 10.10.30.13 MAC address.