New machines on network can't get traffic out for ~5 mins

[duckyduckyducky]

Hello,

I've got a very strange issue on my hands that I could use some help debugging.

My setup:
- OPNsense 18.1.6-amd64
- PC Engines apu2c4: AMD GX-412TC SOC (4 cores) + 4 GB RAM

Any time a new device connects to the network, or reconnects after being off, that device sees the following:
- DHCP address is assigned and network appears configured correctly.
- Can ping router and get responses, but web GUI immediately fails with "connection refused".  Same behavior for both IPv4 and IPv6 addresses.
- Can communicate with other devices on the LAN normally.
- The majority of HTTPS webpages will not load - they stall at "connecting..." in the browser.  Some pages do work though - e.g. google.com is accessible and slashdot.org is not.  Unencrypted HTTP traffic also works for some pages but not others.
- DNS does not work.
- Strangely I _can_ ping google.com, as well as use the website.
- I cannot connect to an outside server via ssh directly by IP address, either IPv4 or IPv6 addresses.  Connection eventually times out.

In all the above cases where the computer could not connect, another computer right next to it which had been on the network for longer was able to make the connection without issue.  After about five minutes everything goes back to normal and all the tests above work on both computers.

I was watching the firewall traffic on the working computer and saw connections being passed from the newly connected computer when trying to load pages that stalled, suggesting to me that it's not an issue with firewall rules.  (Using the web GUI here at the same time that the GUI is not accessible on the other machine.)

I experience the issue on macOS, iOS, and Ubuntu devices, both with DHCP and with static IP addresses configured, over both the wireless AP and via ethernet.  All devices connect to the LAN interface on the apu2 via an unmanaged gigabit switch.

The issue was not present until I moved to the OPNsense router - had been using a consumer box previously.  Thought it might be something IPv6 related, since I don't think my old router was set to use it, but I'm seeing issues with both IPv4 and IPv6 traffic, and the issue of not being able to get to the OPNsense GUI is especially bizarre.  No indications that it's an upstream problem either, as other machines on the network that have been connected for longer can get traffic out just fine.

In some cases, some magic combination of disabling and re-enabling network interfaces repeatedly and spamming the "Renew DHCP Lease" button will reestablish connectivity, but in other cases it doesn't.

¯\_(ツ)_/¯

Thanks, any ideas would be much appreciated!

[duckyduckyducky]

A little more information - testing using my Windows 7 VM under VMware Fusion (OS X host) gave some interesting results.
- With networking in NAT mode, where VMware proxies all the traffic and the VM shares an IP with the host, I can't make any connections at all under the VM.
- With networking in Bridge mode, which gives the VM its own IP address, I could cause the connection issues to start occurring again by just switching between using the host and the VM.  Open a page in the VM, it starts to load, then both host and VM get stuck.  After five mins, VM can load pages.  Try to load one on the host, same effect, both sides stall - after 5 mins I can load pages on host again.

Starting to wonder if this could be a hardware or driver issue with the apu2.  Will try shortly eliminating the ethernet switch and plugging straight into the OPNsense LAN port too, just to make sure it isn't the switch causing the problem.

[duckyduckyducky]

Further update:  I installed OPNsense on a desktop with two NICs and configured equivalently to the apu2, switched it in as the router, and the issue has disappeared. 

At this point I'm assuming it's a hardware or driver issue with the apu2 board.  Anyone else had success running OPNsenese on such a board?  Is anyone aware of driver issues with Intel i210AT NICs?

Here's the board I was using:  https://www.pcengines.ch/apu2.htm

[bitman]

on your apu2 do you have the nic's in auto negotiate mode? it might be that there is some problem with link negotiation?

just a shot in the dark but you never know

[marjohn56]

Further update:  I installed OPNsense on a desktop with two NICs and configured equivalently to the apu2, switched it in as the router, and the issue has disappeared. 

At this point I'm assuming it's a hardware or driver issue with the apu2 board.  Anyone else had success running OPNsenese on such a board?  Is anyone aware of driver issues with Intel i210AT NICs?

Here's the board I was using:  https://www.pcengines.ch/apu2.htm

i use an APU2C4 for testing, never had an issue with it at all. I've also had it running as a 'Live' firewall for a day or two whilst I was tinkering with my Qotom unit, also no problems during that time.

[duckyduckyducky]

Thanks for the replies.  I had seen other positive reports about the apu2 which is why I decided to try it originally, I feel like there should be a way to solve it.

I am most likely running the interfaces in auto-negotiation mode, since I don't recall changing this setting.  Assuming you mean the link parameters and not just DHCP?  Are there known issues with the apu2 or drivers for those Intel NICs?

Forgot to mention before, but another important data point is that I believe I experienced the same issues when I tried running the Sophos XG Home firmware, though I hadn't characterized the issue then.  I thought Sophos was just a bit heavy for the apu2 but looking back all the symptoms are the same as the NIC issue.  Since Sophos is Linux-based it suggests to me something hardware-related rather than driver-related, though I don't know if there are substantial differences between the Linux and BSD drivers.

[marjohn56]

There was a tweak in the settings for APU2's under pfsense and it applies to Opnsense too, that had something to do with the NICs, bit I'm buggered if I can remember what it was...


I'll try and remember, but it may take me a while.