Research Internet connectivity issue

[verasense]

During the last two days I went to bed with Internet access and I woke up without it. My question is: How can I research what is happening?  What should I look for and where? I looked into logs but I could not find anything strange.

Symptoms:
- The router is operating but devices have no Internet access
- However, devices can access machines within the local VLANs, so my computer can see data from the camera, locally
- Cannot access the router GUI by DN but I can by IP
- From the router I can ping google.com and get DN for www.google.com
- I can connect externally via VPN
- When Windows tries to fix the issue by reseting DHCP, I lost the assigned IP (DHCP not working?)
- I had to restart the router to get everything back. I found out that restarting some services a couple of times seems to work too.


Some context:
It all started when I rebooted the router after many months of uninterrupted operation and I upgraded Opnsense to the latest version:
Type   opnsense   
Version   21.7.6   
Architecture   amd64   
Flavour   OpenSSL   
Commit   acdaa7649   
Mirror   https://pkg.opnsense.org/FreeBSD:12:amd64/21.7   
Repositories   OPNsense   
Updated on   Wed Dec 8 13:52:49 UTC 2021   
Checked on   Sat Dec 11 13:29:34 UTC 2021

All packages are up to date:
Your packages are up to date.

I only see this issue, but I don't see the reason, and it seems to work fine:
os-dyndns (misconfigured)   1.27_1   173KiB   OPNsense   Dynamic DNS Support

WAN connection is OK:
   Name   Interface   Protocol   Priority   Gateway   Monitor IP   RTT   RTTd   Loss   Status   Description   
      WAN_... (active)   WAN   IPv4   254 (upstream)   x.x.x.x      ~   ~   ~   Online   Interface


From a client computer I get no DNS access:
> tracert google.com
Unable to resolve target system name google.com.

> nslookup www.google.com
DNS request timed out.
    timeout was 2 seconds.
Server:  UnKnown
Address:  10.1.1.1

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
*** Request to UnKnown timed-out

But from the router I can ping and receive DNS responses with no problem


It seems to me that I could be a problem related to DHCP or DNS, but what should I look for, and in which GUI option, to find the source of error?

Thanks

[bugvito]

I experienced similar issues with 21.7.6.  Reverting to 21.7.5 solved my issues.
I am running on proxmox(intel), with virtio networking.

At times all appeared fine for a day or two, while sometimes the issue would present itself after only a few hours.

Since the issue appears unpredictably and I need a solid connection for work, I wasn't able to look into it with trial and error, while I didn't see anything in my remote logs, and local logs are kept in tmpfs (lost on reboot).

The only packages listed for update would be OpenSense(21.7.6), os-dyndns (1.27_1), and suricata(6.0.4).
This isn't of much help, but if nothing else, may confirm that there's an issue present.

[verasense]

Thanks. It's useful to know that the issue is happening to other people...
I have not seen anything unusual though. I will keep checking logs, but if anyone has an idea of where to look and for what, it would be appreciated.

[autone]

The issue is most likely Suricata. Disable intrusion detection. See if that helps.

[verasense]

I am not sure this is the source of the issue but you are right there is something odd there. Yesterday I found Suricata was stopped for some reason, and the logs were saying:

Code: [Select]

2021-12-12T07:04:15 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.telnet.busybox' is checked but not set. Checked in 2023019 and 2 other sigs
2021-12-12T07:04:15 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'et.DocVBAProject' is checked but not set. Checked in 2020170 and 0 other sigs
2021-12-12T07:03:48 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-user_agents.rules:250 uses unknown classtype: "pup-activity", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:44 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-malware.rules:15756 uses unknown classtype: "credential-theft", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:41 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-malware.rules:9890 uses unknown classtype: "social-engineering", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:41 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-malware.rules:8962 uses unknown classtype: "external-ip-check", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:39 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-info.rules:694 uses unknown classtype: "coin-mining", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:37 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-exploit.rules:800 uses unknown classtype: "exploit-kit", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:37 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-dns.rules:112 uses unknown classtype: "targeted-activity", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:37 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-attack_response.rules:488 uses unknown classtype: "domain-c2", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:37 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-activex.rules:788 uses unknown classtype: "command-and-control", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:02:25 suricata[4057] [100200] <Notice> -- rule reload starting
2021-12-11T13:44:20 suricata[4057] [100200] <Notice> -- all 5 packet processing threads, 4 management threads initialized, engine started.
This might be a misconfiguration from my side but I cannot see where. All rules are updated and enabled.

Not sure what I have to modify to have Suricata working... Disabling it is a workaround, but not ideal.

I was checking similar issues and found some posts, but nothing useful...

[bugvito]

21.7.7 is out. This will change testing outcomes, but I'm locking suricata, and upgrading to 21.7.7.

I'll test this way for a few days, and if there's no issues, I might unlock suricata, and wait a few more days.

[bugvito]

Unfortunately after a day or so on 21.7.7 (keeping suricata from 21.7.5) I started having issues with anything voice related (Microsoft Teams for example), but no network cutout like before.  Restarted everything, same issue right away. Unfortunately I don't have the cycles to look into this.  Reverted to 21.7.5, and all seems fine again, but could also simply be coincidence.  Sorry for not having time to spend on this and gather useful details.

[verasense]

In my case, keeping 21.7.6 with Suricata off (it was stopped automatically, without my intervention, probably something went wrong) I have not experienced the issue again in the past days. I think that for some reason some of the services were probably shut down automatically and that is why I experienced the DHCP/DNS problems.

I am still not sure how to research the source of the problem, but at least it is not repeating itself.

[cookiemonster]

Start with basics and make no assumptions like version Y is broken and going to version X everything is fine. It'll take you nowhere near finding a solution. That is because you don't know what the problem is, which in fairness to you, is what you said from the start. It might be another problem unrelated to yours in the first place.

During the last two days I went to bed with Internet access and I woke up without it

Break the problem down. Take a client you can interact with like a pc. Does it have name resolution working? Has it got an ip, does it appear in the firewall traffic when you attempt to get through it?
You need to be ready to interact with this client and OPN at the same time, hence a PC is good.

When Windows tries to fix the issue by reseting DHCP, I lost the assigned IP (DHCP not working?)

Maybe I misunderstand but that is what is meant to happen. It says, hey dhcpd server, I want a new ip please, which one should that be? Gets one. Unless it has been reserved at the dhcp server, it might get the same or another one.

Finally, describe your setup. So many variables.

[verasense]

After many more days, I can confirm that the problem has not been reproduced again.

So I was not able to see the problem again or to find ways to research the issue. I think it was something making DHCP and DNS services go down.

To be clear (especially wrt the previous post), you are referring to the other person, I never said this was a version problem. I did not know what the problem source was but I DO know it started after I rebooted the router in a long time after a version update -- BUT this does not mean that it is a problem of the new release, I do not know. I was just looking for ways to research what the problem was.

This:
 

Code: [Select]

  os-dyndns (misconfigured)   1.27_1   173KiB   OPNsense   Dynamic DNS Supportis solved via:
https://forum.opnsense.org/index.php?topic=22529.0#msg111227

"System: Firmware: Status", bottom right: "Resolve plugin conflicts" option "Reset all local conflicts"

The only thing that stays is a Suricata problem, but I think this should be reported in a new forum post.

[bugvito]

I'm glad the that issue is resolved for you!

For the "version" discussion; whether it's a configuration that no longer jives with an updated package, or a package that is "broken", old definitions causing issues with Suricata, or any other case, coming only after an update and can be repeated, this is a versioning issue.  This does not mean that a package is broken per se, but a set of condition lead to an issue with some package version.

With verasense no longer experiencing the problem, I'll take a deeper look time permitting.

[bugvito]

I updated to 2.7.7 with suricata 6.0.4, and the network drop issue reappeared right away.  Restarted the OpnSense VM, same issue.

I then disabled suricata, and no issues at all.  I then re-enabled IPS (suricata), and no issues, aside from wifi calling and Teams not letting me talk, but I could hear just fine. Looking into it, a rule "Conficker-C P2P encrypted traffic UDP Ping Packet" was blocking outbound UDP traffic on port 4500; setting this to "Alert" only resolved this issue.  This last issue is not directly related to this post, but seems to support the suggestion that something changed in the behavior of suricata, as I had not added any rules and had updated to the latest rulesets in all cases.  This rule had been configured the same way in all cases, and only started to cause issues after moving to suricata 6.0.4.

I'm not a fan of the "solution", as disabling and re-enabling a service does not resolve the actual problem, akin to a duct tape solution, but I am not in a position to look under the hood and see if a caching issue, configuration change between version, etc... could be the cause.

All seems good now on my end.

[lnaza]

https://en.wikipedia.org/wiki/Conficker

Conficker D    2009-03-04

HTTP pull
        Downloads daily from any 500 of 50,000 pseudorandom domains over 110 TLDs
P2P push/pull
        Uses custom protocol to scan for infected peers via UDP, then transfer via TCP

Blocks certain DNS lookups
      Does an in-memory patch of DNSAPI.DLL to block lookups of anti-malware related web sites
Disables Safe Mode
Disables AutoUpdate
Kills anti-malware
        Scans for and terminates processes with names of anti-malware, patch or diagnostic utilities at one-second intervals
Downloads and installs Conficker E