[Solved] OPNsense loses connectivity for some minutes but other devices don't

[fiatjaf]

I'm running OPNsense 20.1 on normal consumer hardware. I've been doing it for about 2 years, without issues.

My setup is like this:
1. ISP provides me with a modem/router
2. An ethernet cable goes from the modem/router to my OPNSense machine on the WAN interface
3. From the OPNSense machine LAN interface another cable goes to a switch that connects other computers in the lan

Recently something odd started happening:
 - Machines in the lan would suddenly lose connectivity. After some time it comes back, then falls again, and the cycle repeats an indefinite number of times. It happens mostly in some parts of the day, but it's not a certain thing.
 - OPNSense seems fine.
 - If there's another machine connected directly to the same ISP's modem/router that machine still has connectivity and all is fine.
 - If I take the plug out (from the OPNSense---modem/router cable) and put it again connectivity immediately returns to the lan (only to fall again some seconds or minutes later).

I've changed cables, ethernet boards, the OPNSense machine, the modem ethernet port, but it keeps happening, so I imagine maybe it was an issue with OPNSense itself? Am I crazy? What is this? Is my ISP trolling me?

[Supermule]

Are you doing double nat'ing since you run OpnSense behind another router?

[fiatjaf]

Yes, I am.
I shouldn't, right?

[Supermule]

You shouldnt... get it behind the modem (transparent) and use the public IP.

How many public IP's do you get from your provider? If only one, then thats the reason why it switches...

[fiatjaf]

What do you mean it switches? It's one public IP, fixed. It never switches. I just lose internet connectivity on OPNSense but I have no idea why.

I think my ISPs modem/router doesn't allow me to connect directly. Years ago, when I had a different modem from the ISP I remember connecting using PPPoe, but now they have their own firmware in the modem and it's very restrictive. Does this make sense? Should I get my own modem and throw this modem from the ISP in the trash? Do you think this could be related to my issue?

[Supermule]

I read it wrong sorry.

[FingerlessGloves]

What do you mean it switches? It's one public IP, fixed. It never switches. I just lose internet connectivity on OPNSense but I have no idea why.

I think my ISPs modem/router doesn't allow me to connect directly. Years ago, when I had a different modem from the ISP I remember connecting using PPPoe, but now they have their own firmware in the modem and it's very restrictive. Does this make sense? Should I get my own modem and throw this modem from the ISP in the trash? Do you think this could be related to my issue?

Who's your ISP?

how long is the DHCP lease when you plug it in to your laptop say? I've heard things were the ISP do weird things with DHCP lease times, then you lose connection if you don't send another DHCP request after so long.

But if the ISPs Router is then connecting to OPNsense, then that wouldn't be the case. What mode is the ISP router in?

[fiatjaf]

I'm in Brazil, my ISP is Vivo.

The modem has a DHCP server. OPNSense gets an IP from it. I can't control any specifics of it, only generic firewall stuff. The lease time on my machine and on OPNSense from the modem's DHCP server is 43200 seconds it seems.

Now I realize I should be looking at logs on OPNSense to try to get some idea of what is going on. I'm not sure where to look, but will try.

[fiatjaf]

Ok, I think I found out the problem, but I can't find the solution.

On OPSense I found this:

Code: [Select]

~> dmesg
arp: 60:1d:91:50:69:31 is using my IP address 192.168.15.192 on bge0!
arp: d4:63:c6:b0:ba:31 is using my IP address 192.168.15.192 on bge0!
arp: 60:1d:91:50:69:31 is using my IP address 192.168.15.192 on bge0!
The appearance of these lines coincide with the times the connection breaks.

I don't know what these MAC addresses are, but I tried fixing the IP of the OPNSense machine at the ISP modem (before it was 192.168.15.2) to 192.168.15.192 and minutes after the same messages appear again, the same 2 MAC addresses. Either someone is trolling me or there is something badly wrong happening somewhere.

What can I do to prevent this?

[stefanpf]

Sorry, I've no idea but would suggest 1st to find out to which device these mac addresses belong.

The Range of both are registered to
Motorola Mobility LLC, a Lenovo Company

- Check the macs at your Sense Box
   Interfaces / assignments
- Check the mac of your admin PC/Notebook
  (the device that you put into the Modem for testing)
- Check the mac of your Modem
  Put your Laptop into the Modem, get an IP and Ping the IP of the Modem. After that you enter "Arp -a" (If you use Windows)at the command line to lookup the mac of the modem.

[fiatjaf]

None of these MAC addresses match, but it does match against a rogue MAC that was sending DHCPDISCOVER messages through the lan to my OPNSense box, probably through WiFi. Probably the other too, but the logs only go as far as a few hours ago (are there archived logs somewhere on OPNSense?).

So it means it's someone's phone? That would explain why it happens only at some times of the day: the person sees my WiFi and tries to connect to it. Since it doesn't get a DHCPOFFER from OPNSense (which would be in the 192.168.13.* range) it self-assigns 192.168.15.192 to itself for some reason and its packets end up reaching the ISP modem at 192.168.15.1 which in turns causes OPNSense to disconnect?

The WiFi is open and the only security I have in place is a static ARP table (192.168.13.0-254) on OPNSense (I know this is not optimal and that I should change it, but it's there for historical reasons).

[stefanpf]

You're not serious, are you?
You bare your windows while leaving the gate open and ask yourself if someone is trolling you 

[fiatjaf]

Indeed, that was the issue. For some reason either the modem was giving the same IP address it should have given to OPNSense through its own WiFi interface to a rogue cellphone. I believe this is a bug in the stupid ISP modem firmware as it did that and shown it in its own dashboard as if everything was ok.

Solved by disabling the modem's WiFi interface, but maybe it could also work by giving a static ARP entry for the offender and then blocking it on the modem's firewall.