Packet received by interface but blocked before firewall

[Somnolus]

I'm trying to pass a very specific UDP packet from one interface to another on different networks on the same router.  I've run a packet capture on both interfaces, I can see the packet coming in on the input interface but it just disappears after that.  There are no firewall logs indicating the packet was blocked.  I've tried sending the packet through with the firewall and NAT disabled but it still doesn't get processed.  I'm convinced its either the interface or Kernel blocking the packet before the firewall can process it.

If anyone has any tips on where I can look to see why the packet is being blocked that would be great.  Any help would be appreciated here as I'm at a complete loss on where to troubleshoot next.

[viragomann]

So should the packet even go out on the desired interface according the routing table?
Or did you forward the traffic to a device connected to this interface?

Maybe we can get closer to the issue if you give some more details: destination IP, networks, routing table, rules.

[nero355]

I've run a packet capture on both interfaces, I can see the packet coming in on the input interface but it just disappears after that.

You should share your tcpdump/pcap output IMHO if you want anyone to say anything useful about it :)

[lmoore]

There are no firewall logs indicating the packet was blocked.

The likelihood is that it is being blocked by a firewall rule, or you don't have a rule to allow it out of the destination interface. It's difficult to suggest anything without seeing the relevant rules and if Source or Destination NAT is required.

Ensure you have enabled logging of Default block and Default pass in Firewall -> Settings -> Advanced.

In addition, if you have configured any block rules, ensure logging is enabled in them so you can spot anything amiss.

[Monviech (Cedrik)]

What kind of network device are you using (intel nic)?
Are VLANs involved?
How large is the UDP packet, would it need to be fragmented?
Is intrusion detection used (netmap driver)?
Is a Captive Portal configured? (ipfw enabled, can block packets before pf)
Is a ipsec policy installed that might blackhole this paket?

I know a few things that can make packets "vanish" without a trace in tcpdump, the above are some.

[Somnolus]

I have 2 devices that need to establish communication between eachother.  each device is on its own network, but connected to the same router.  These are separate physical connections, no VLANs are involved.  The machine is a intel N305 router with 6 2.5gb ports using the intel i226-v chipset.  The machine is running proxmox with 5 of the 6 ports directly passed through to opnsense.  Both devices are on ports directly passed through to opnsense.

Device 1 is on network 10.0.0.0/24, Device 2 is on 192.168.4.0/24.  Both machines have been given static IP's via KeaDHCP.
Machine 1 is set to ip 10.0.0.55, machine 2 is assigned ip 192.168.4.27

Machine 1 is a server that handles file sharing, Machine 2 is a system that requests the files.  Communication occurs over 2 ports.  Port 62966 and 62967.  Machine 2 broadcasts a multicast request over port 62966 that hits the router and runs through UDP-Broadcast-Relay which makes its way to machine 1.  machine one than responds to machine 2 letting it know its IP and the port to use for data transfer.  Everything works as it should both machines see each other just fine and all communication passes normally.  the problem is when machine 2 sends a UDP packet requesting data from machine 1.  That packet hits the router interface but doesn't seem to go anywhere beyond that.  I've enabled firewall logging for everything including NAT rules under the advances settings. I don't have captive portal, or ipsec enabled.
I've attached packet captures that were run directly inside opnsense.  The packet in question is under interface 2 and it repeats 4 times.

[lmoore]

The packet in question is under interface 2 and it repeats 4 times

The node at 192.168.4.27 is broadcasting the reply. For the packet to flow through pf in OPNsense, it would need to be a Unicast address forwarded to a8:b8:e0:04:91:1b.