Two devices will not stay connected

[Sage_viper]

Hello,
I'm pretty new to OPNsense and just trying to wrap my head around more networking information, so bare with me.

I recently switched to OPNsense from a Netgear router that was having some issues. I have things pretty well dialed in without doing anything fancy at this point, but there is one thorn in my side.

All of my devices connect without issue to the network (mostly over Wifi on a dedicated AP) except for two HP work laptops. After a lot of trial an error, I thought I narrowed down the problem to IPv6, after I turned it on for the LAN interface (I have admin to one of these laptops, but not the other - so I can't disable on the adapter, though this didn't make a difference on the one). However, after one day without issues, they returned the next day: I can connect to the Wifi, but only with local network access, no internet. What REALLY fixes this, temporarily, is making some kind of minor change to the LAN interface, saving, and applying the change. Whether this is turning IPv6 on or off, or even changing IPv4 from Static IPv4 to DHCP (which fails due to a DHCP server running message) and back to Static IPv4, so ultimately making no change at all.

Has anyone run into this? Does anyone have any advice on what to look for? I'm not sure what kind of setup info would be best to share, so let me know what may be helpful to share here, or to look into.

Thanks in advance to anyone who can help me tackle this!

Edit: Here are some basics:
Version: 25.1.3
Network setup:
WAN: PPPoE (CenturyLink Fiber) over motherboard's onboard Intel LAN
LAN: Static IPv4 over Realtek PCIe Ethernet adapter
I have a managed switch plugged into this, with a TP-Link AP connected, as well as a handful of other hardwired devices.

[EricPerl]

Realtek NICs don't have a very good rep under FreeBSD/OPN...
The fact that resetting the network on that NIC fixes things might point in that direction.

This said, I don't know how likely it is that it only affects 2 machines.
Can you still ping the LAN address from the work machines? from others when the work machines are down?

[cookiemonster]

My initial thoughts too. Realtek = troubles in freeBSD.
That said, it could be about anything at this point, dns or an ipv6 when is not required by the setup.
I'd start by looking for clues in dmesg.

[Sage_viper]

I can still access the local network, they just won't connect to the internet, even though every other device on the LAN works no problem, connected the exact same way.

It would be odd that only these two devices have problems due to the NIC, but it's only about $20-30 to find out I suppose.

[cookiemonster]

those are symptoms only and points to perhaps the NIC not yet being the source of the problem indeed.
Any clues in dmesg then?

[Sage_viper]

A few entries where the LAN goes down then back up, and some items that say different MAC addresses are "using my IP at 192.168.1.1", one of which is my Tuenas box which is hardwired, another doesn't show up in my leases.

None of these have timestamps, though, so I'm not sure when they happened in relation to the errors. I'll have to wait a bit until they drop off the connection and I can pull dmesg immediately after. I'll keep looking through to see if anything sticks out, though. Appreciate the direction!

[cookiemonster]

LAN goes down and back up is the bit we were hoping not to find. Now as to what causes it is what we're looking for. As you can begin to see, it is possible that the nic is faky. First of all, try to use the vendor driver if you aren't already.
Then

some items that say different MAC addresses are "using my IP at 192.168.1.1"

is not a good sign. It might be what triggers the problem or not but you should fix that nonetheless.

[EricPerl]

Given that's the default GW IP of LAN, it could very well be the problem.
If another host claims that IP...
The ARP table on the work machines when they fail should point at the offender.

[Sage_viper]

So I looked more into that IP issue.
It is my TrueNAS box, which has a static IP set on the box itself, as well as a reservation in OPNsense. It also moved from one MAC to another at one point, but again I'm not sure what the timeframe is on this - I was messing with that device a week or so ago, so it could have been related to that.

Code Select Expand

arp: a8:a1:59:65:5d:68 is using my IP address 192.168.1.1 on re0!
arp: a8:a1:59:65:5d:68 is using my IP address 192.168.1.1 on re0!
arp: 192.168.1.44 moved from a8:a1:59:65:5d:68 to 22:9e:2e:da:d3:2d on re0
tailscale0: link state changed to DOWN
tun0: link state changed to UP
tun0: changing name to 'tailscale0'
tailscale0: link state changed to DOWN
tun0: link state changed to UP
tun0: changing name to 'tailscale0'
re0: link state changed to DOWN
vlan01: link state changed to DOWN
re0: link state changed to UP
vlan01: link state changed to UP
arp: 22:9e:2e:da:d3:2d is using my IP address 192.168.1.1 on re0!
arp: 22:9e:2e:da:d3:2d is using my IP address 192.168.1.1 on re0!
arp: 22:9e:2e:da:d3:2d is using my IP address 192.168.1.1 on re0!
arp: 22:9e:2e:da:d3:2d is using my IP address 192.168.1.1 on re0!
arp: 22:9e:2e:da:d3:2d is using my IP address 192.168.1.1 on re0!
arp: 22:9e:2e:da:d3:2d is using my IP address 192.168.1.1 on re0!

I was just able to observe one of the devices failing to connect to the internet, then after removing DHCPv6 and applying, observing it reconnect, there doesn't seem to be anything different in dmesg. I installed the vendor driver when I was installing OPNsense initially but it didn't show in the Plugins section, so I just ran it there to be sure and it looks like it was indeed installed. I may at this point just find an Intel based NIC and go from there.

[EricPerl]

I assume you're not setting the static IP of these machines to the LAN IP.
It's weird enough that one machine would claim it. 2 of them???
You might want to double check your DHCP config.
The first MAC indicates an ASRock machine. The second one seems unknown.

[Sage_viper]

I assume you're not setting the static IP of these machines to the LAN IP.
It's weird enough that one machine would claim it. 2 of them???
You might want to double check your DHCP config.
The first MAC indicates an ASRock machine. The second one seems unknown.

No, definitely not. So the first MAC is the actual MAC of the ASRock motherboard running my TrueNAS system, the second is the generated MAC from the static IP I created when I made a virtual bridge within TrueNAS that I created with the static IP, so that VMs can grab their own IP. I can't see a clear reason why it would be trying to grab the gateway IP.

I've ordered an Intel NIC that is coming today, so we'll see if that helps iron out what is going on.

[EricPerl]

The fact that 2 MACs related to your TrueNAS setup seem to be involved is a red flag for me.
The mention of a bridge adds up.
I would take a look at its status next time your network is in trouble.

[Patrick M. Hausen]

If you use a separate NIC in TrueNAS for VMs and you (correctly) preconfigure a bridge interface for that and if the VMs are supposed to be in the same IP network as the other (main) TrueNAS interface, then you must not assign an IP address to that bridge interface. It's a layer 2 connection for your VMs only. You cannot have two interfaces with different IP addresses in the same network.

[Sage_viper]

I currently only have one NIC in my TrueNAS box but I followed the instructions from this video to set up a static IP and bridge for the single NIC: https://www.youtube.com/watch?v=uPkoeWUfiHU
So basically assigning a static IP on the physical interface, then removing said IP, creating a bridge, and assigning that static IP as an Alias on the bridge so the static IP remains. However, I suppose TrueNAS ends up creating a virtual MAC for this connection internally and no longer uses the physical MAC. This is the only Alias I have set up for the device. The two VMs I have running have already grabbed new, unique IPs from the DHCP range, which I then created reservations for in Opnsense.

This was before Opnsense - and it has worked so far, more or less. Are there any alternate resources one might suggest for a better way to set this up?

[Patrick M. Hausen]

However, I suppose TrueNAS ends up creating a virtual MAC for this connection internally and no longer uses the physical MAC.

Yes, and this generated MAC will change at every reboot at least for TrueNAS CORE (FreeBSD). Don't know about Linux.

But if it is TrueNAS CORE you can do something to fix it - go to System > Tunables and add a tunable following the screen shot:



The bridge should now have the same MAC address as the physical member interface on every reboot.