Strange Networkproblems

[hongo]

Here is my setup:

1 Port -> WAN (configured with DHCP)
2 Ports -> bridge0(LAN)
2 Ports -> bridge1(WIFI)

Wifi-Ports:
  - igb8 connected to a NAS
  - igb9 connected to an access-point

All devices in Wifi are configured with DHCP and have static arp entries.

Sometimes, Ports "die" on the WIFI-Interface. This means, the device connected on this NIC-port is not available. The NIC-Port is marked as UP on both sides but if i do a tcpdump on the firewall and on the device-connected on this port, and try to make an arp-scan from the firewall to the device, then i can see arp-packets goint to my device, on the device I see that it replies correctly. But I can't see any replies on the tcpdump of the firewall.
That is strange, because the device is connected via ethernet-cable(Yes i also tried another cable too). And the problem is fixed as soon as i disable the WIFI-Interface and enable it again. The problem doesn't always occur. Somedays it doesn't occur at all, sometimes it happens many times.

If a device "died" i tried out some things:

- disconnect the device from the firewall, and directly connect it to another device. no problems.
- disconnect the diveice from the firewall and connect another device on the same port. same problems

I have no idea how to fix this or even how to find out the source of this strange behaviour.

[elektroinside]

Sounds to me that your clients are not always identified (which is strange indeed, should not happen). If you enabled static arp entries in DHCP, the mac address is crucial. If for some reason that changes, your clients won't work...

Also, if once connected to the WIFI you are behind the OPNsense box and you're double NAT-ing, that could cause some issues.

[hongo]

The NAS-Box, which was lately affected by this bug, is connected via cable and not wifi.  And even if it would be connected, the wifi is configured as access-point, not router. So there is no nat in this network.
The mac-adresses do not change. I checked it and can also see it in my tcpdump(on the client device).

What i don't understand is:
client sends ARP-Reply to the firewall. I see the arp-packet in the tcpdump on the client, but not in the tcpdump on the firewall. So even if the packet gets dropped by the firewall i expect to see that packet. Or am I wrong? 

[elektroinside]

If I'm not mistaken, if you don't have a static arp entry for a client, you shouldn't even get an IP from the DHCP. So connections could not be possible at all. I'm guessing, that if you configure your clients to get an IP from the DHCP (and not use static IPs), you are getting one.. right?

[hongo]

Right. DHCP is working

[elektroinside]

In this case, I think your firewall is silently dropping packets for some reason. I see no other reason why you can't see them. Do you have IPS with IDS enabled by any chance? If so, with 18.1.1 you should be able to see all packets blocked by IDS.

[hongo]

I deaktivated IDS now. It shouldn't block anything because I didn't configure IPS, but who knows. Since I am out of ideas this is my last try. If the problem occurs again, I will setup a Linux-Firewall. If this happens then it might be a hardware error or user-fail but if its fixed then, I'll let you know.

[elektroinside]

Also, if you're using aliases with firewall rules, don't forget to check the "log this rule" (something like this) option in the rule, otherwise, you will not see the blocked connection...

[hongo]

Thanks but I don't have any explicit block-rules.

[hongo]

Problem solved using Linux(with bridge, dnsmasq, shorewall, suricata, aso.) instead. Seems like it isn't a hardware problem. I would really really have known what the cause was.

[elektroinside]

Thanks for getting back on this. I'm sorry OPNsense didn't work out for you.

[hongo]

I won't give up. I created an Image of my Opnsense-Installation and will try again to debug it as soon as I can make my network offline for a while..

[elektroinside]

A lot of things changed sinced then. Maybe you could try the latest 18.1.5?