Hi
I have an issue where I see traffic come in on one interface of the firewall but then it does not go out on the another interface. I have permit rule for the traffic but still I do not see traffic on destination interface of the firewall. Any help will be much appreciated. Below is scenario:
client---->if1(opnsense)------>if2(opnsense)----server
Here client traffic is seen on if1 of the opnsense firewall and server is on same network as if2 of the opnsense. There are no drop logs and I have a permit rule so it is not permissions issue but for some reason I do not see traffic on if2 of the opnsense.
I did packet captures and I see packets on if1 but not on if2.
Thanks
Maybe you had if1 first as the preconfigured WAN interface and the default NAT rules are still in place? NAT rules are prioritzed higher than normal firewall rules.
I am not sure what WAN interface means but this is sub-interface for a VLAN and no NAT rules were configured. This interface was configured to connect to another LAN segment. This interface has gateway attached to it. So basically I try to connect from a client to a server behind this firewall.
client traffic enters firewall on if1(with gateway attached to it) and destination is the server which is directly connected to if2. I see client traffic on if1 but when I capture packets on if2 I see no traffic. At this point I am only concerned about traffic from client to server.
When you first install OpnSense, it has a default setup for two interfaces, one for WAN with NAT rules and one for LAN. There are some default rules in place to make life easier for beginners, like an "allow any" rule for the first LAN interface.
But never mind, just inspect Firewall: NAT: Outbound. If there are any automatic or manual rules for your if1, you have identified the culprit, because you cannot make a connection against an outbound NAT rule (other than creating a DNAT rule).
sure will inspect the NAT rules. But I am not clear if outbound NAT rules will impact traffic inbound on same interface. The connection is initiated from a client that is received by firewall on if1(inbound) and target is a server directly connected to if2. So if there are any outbound NAT rules on if1 that should not match traffic inbound on if1.
Correct so far. But all response packets for your inbound traffic will go through NAT. So even if you allow inbound traffic, answers will not come from the server, but from the if1 interface address. Therefore, that still does not work. You would need DNAT rules for that.
The way you set this up, you obviously want to use OpnSense as a pure router between two arbitrary subnets, yet you may have NAT in place, which is a hack to connect non-routeable networks with the global internet. That won't work (if my guess is right).
Correct I would like opnsense to behave like a router with no source or destination translation. I checked there are outbound NAT rules but for interface which is my gateway to internet and not if1. But again:
If with outbound NAT on an interface the traffic received will be source NAT'd but destination would remain same. So in my case destination is server on if2(same network) so I should still see packets on if2(packet capture) with source as NAT'd IP on if1 and destination as target server.
The interface with the GW to the internet is typically WAN (by default).
Also by default, the 2nd interface is LAN. Then OPT1 and so on.
Assuming both systems have IP on interfaces with distinct subnets, logging enabled on all rules, you should see:
IN traffic on IFL1, OUT traffic on IFL2 for the request.
The IN on IFL2 followed by OUT on IFL1 for the replies.
The FW live view should show you the request part. You have to do packet captures to see the replies.
When the interfaces are set up properly, you typically don't have to do anything to get routing to work.
You might want to share that part.
OK here is what I see:
With packet capture on if1 and if2, I see client to server requests on if1 but not on if2. In live view I see traffic IN on if1 but no OUT on if2 so it is sure packet arrived and was logged by if1 but thereafter it got lost somewhere between if1 to if2 within the firewall itself. If it helps I can snip configurations on if1, if2, NAT, firewall rule for interesting traffic here. I can also snip packet captures.
It seems likely that you don't actually have a firewall rule to allow the connection that you're attempting. Could you show the rule that you think should allow it (and state which interface the rule is on)?
I am not able to paste the rule on interface if1. Is there way to paste ?
You should be able to attach an image (screenshot).
Please find attached firewall rule on interface that receives the traffic from client
Reading back through the discussion, you said something that made me pause - "So in my case destination is server on if2(same network)". What do you mean by "same network"? You can't have two interfaces on the same network (and expect things to work)...
The target server is on same network as if2. In a sense server is directly connected to if2(same network). Why won't this work ? It is similar to having two hosts on same network so in this case one is server and other is the interface if2 on firewall. This makes them both part of the same broadcast domain and thus need no explicit routing to talk to each other.
Now when traffic from client enters firewall on if1 packet would be like:
src: 10.28.140.50
destination: 10.10.30.13
if1: 10.28.140.49
if2: 10.10.30.2
In this case when firewall sees destination to be 10.10.30.13 it knows it needs to send it to if2 and then ARP broadcast on if2- looking for 10.10.30.13 MAC address.
OK, I thought you might be saying that "if1" and "if2" were on the same network.
Screenshots of [Interfaces > Overview] and [System > Routes > Status] might be helpful, along with the source and destination IP addresses.
Attached are screenshots that may help.
Last of the screenshots
Most of the screenshots are too blurry to read... and they're not the ones I asked for anyway.
Do you have [Firewall > Settings > Advanced > Logging > Log packets matched from the default pass rules] enabled? If not, you won't see anything logged outbound (unless you have a specific outbound rule, with logging enabled)
Would I still not see them in packet captures ? I can see them on if2 captures when traffic is coming back from other interface. So I don't really think it is related to logging enabled or not. If I can see packets in captures on same interface if2 when traffic is returned from internet I should be able to see them if they are actually forwarded from if1 to if2. Isn't it ?
And yes the option to log packets matched from the default pass rules is checked/enabled but I still not see them in live view OR packet captures. I am not sure what am I missing when this should be simple L3 forwarding
It generally is that simple.
Earlier you said "client traffic enters firewall on if1(with gateway attached to it)" - what's the gateway for?
Assuming there's no NAT or policy-based routing involved, the traffic should get forwarded according to the routing table, which is why I suggested a screenshot of that, as well the interfaces overview.
You could also get a shell on the firewall and run `route get 10.10.30.13` to see what interface would be used...
The gateway is to route all default traffic via this interface(if1) and attached gateway as I would like to retire old gateway.
Check the interface config - do you have "Block private networks" and "Block bogon networks" unchecked?
They are unchecked.
Back to the routing table, then.
Also, can you ping the destination host from the firewall?
Yes, if I ping the host 10.10.30.13 from firewall with source interface of if1 I get ping working. This is really strange.
I'm totally confused with regard to the overall topology here.
GW to internet on one interface, GW on IFL1 for default traffic
Why can't we get a screenshot of Interfaces > Overview ?
Whatever was there seems to have been removed...
You can blur parts of the public IPs if you want but the rest is safe.
We're at reply #25 and we don't even know what kind of traffic we're talking about.
attached is overview. I am trying to ping from a host on VLAN 777 to a host on VLAN 30. If you see both VLANs/networks are directly connected to firewall and no explicit routing is required.The firewall rule is a permit for which I can see incoming traffic on VLAN 777 but I see not OUT traffic on VLAN 30 both in live view nor packet captures. I am trying a simple inter-vlan routing here.
I had to resize the image because of upload size limit which makes screenshot blurry but can still show meaningful information.
It's barely readable.
I'm unclear why there are that many gateways.
And you also seem to have manual routes defined on top of the default one for each VLAN.
In particular, there appears to be a 10.10.0.0/16 route on LAN_DEFAULT that encompasses the subnets of many 10.10.X.0/24 VLANs, including the one you target.
What's the output of the `route get 10.10.30.13` command suggested by dseven earlier (#20)?
root@a:~ # route get 10.10.30.13
route to: testhost
destination: 10.10.30.0
mask: 255.255.255.0
fib: 0
interface: igb1_vlan30
flags: <UP,DONE,PINNED>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0
root@a:~ #
Hmm, I just noticed something else:
Per #14:
Quote from: firewall_newbie on March 13, 2025, 10:46:15 AM...
src: 10.28.140.50
destination: 10.10.30.13
if1: 10.28.140.49
if2: 10.10.30.2
...
Now looking at the source interface:
Static IP: 10.28.140.49/28
Gateway: 10.28.140.50
Routes: 10.28.140.48/28 & another
I can't even describe how strange that looks compared to a WAN gateway (trying to access a machine on my private network).
It is not strange at all if you try to dig in the requirement.
I have a host coming in on interface 777 with a target on interface 30. The gateway on 777 was configured to use it as a PBF for a specific host to route via gateway connected on 777.
It is routing and firewall rules that play which I assume are configured correctly. Firewall should already know about source and destination. The static routes configured are less preferred wrt subnet mask where firewall has more specific networks on it with a /24(VLAN 30) and /28(VLAN 777).
So for routing firewall will consider both source(10.28.140.50) and destination(10.10.30.13) as directly connected and forward packets. I don't think having a gateway tied to interface 777 would impact this routing decision.
Upstream or downstream?
The other case where I've had to define a gateway was when I temporarily reconfigured my internal OPN without NAT.
https://forum.opnsense.org/index.php?topic=45558.msg228062#msg228062 (https://forum.opnsense.org/index.php?topic=45558.msg228062#msg228062)
To route traffic back to it, I created a gateway on my edge OPN and manually routed the internal subnets to it.
Is 10.28.140.50 behaving as an internal router for the 10.200.0.0/16 network(s)?