During the last two days I went to bed with Internet access and I woke up without it. My question is: How can I research what is happening? What should I look for and where? I looked into logs but I could not find anything strange.
Symptoms:
- The router is operating but devices have no Internet access
- However, devices can access machines within the local VLANs, so my computer can see data from the camera, locally
- Cannot access the router GUI by DN but I can by IP
- From the router I can ping google.com and get DN for www.google.com
- I can connect externally via VPN
- When Windows tries to fix the issue by reseting DHCP, I lost the assigned IP (DHCP not working?)
- I had to restart the router to get everything back. I found out that restarting some services a couple of times seems to work too.
Some context:
It all started when I rebooted the router after many months of uninterrupted operation and I upgraded Opnsense to the latest version:
Type opnsense
Version 21.7.6
Architecture amd64
Flavour OpenSSL
Commit acdaa7649
Mirror https://pkg.opnsense.org/FreeBSD:12:amd64/21.7
Repositories OPNsense
Updated on Wed Dec 8 13:52:49 UTC 2021
Checked on Sat Dec 11 13:29:34 UTC 2021
All packages are up to date:
Your packages are up to date.
I only see this issue, but I don't see the reason, and it seems to work fine:
os-dyndns (misconfigured) 1.27_1 173KiB OPNsense Dynamic DNS Support
WAN connection is OK:
Name Interface Protocol Priority Gateway Monitor IP RTT RTTd Loss Status Description
WAN_... (active) WAN IPv4 254 (upstream) x.x.x.x ~ ~ ~ Online Interface
From a client computer I get no DNS access:
> tracert google.com
Unable to resolve target system name google.com.
> nslookup www.google.com
DNS request timed out.
timeout was 2 seconds.
Server: UnKnown
Address: 10.1.1.1
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to UnKnown timed-out
But from the router I can ping and receive DNS responses with no problem
It seems to me that I could be a problem related to DHCP or DNS, but what should I look for, and in which GUI option, to find the source of error?
Thanks
I experienced similar issues with 21.7.6. Reverting to 21.7.5 solved my issues.
I am running on proxmox(intel), with virtio networking.
At times all appeared fine for a day or two, while sometimes the issue would present itself after only a few hours.
Since the issue appears unpredictably and I need a solid connection for work, I wasn't able to look into it with trial and error, while I didn't see anything in my remote logs, and local logs are kept in tmpfs (lost on reboot).
The only packages listed for update would be OpenSense(21.7.6), os-dyndns (1.27_1), and suricata(6.0.4).
This isn't of much help, but if nothing else, may confirm that there's an issue present.
Thanks. It's useful to know that the issue is happening to other people...
I have not seen anything unusual though. I will keep checking logs, but if anyone has an idea of where to look and for what, it would be appreciated.
The issue is most likely Suricata. Disable intrusion detection. See if that helps.
I am not sure this is the source of the issue but you are right there is something odd there. Yesterday I found Suricata was stopped for some reason, and the logs were saying:
2021-12-12T07:04:15 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.telnet.busybox' is checked but not set. Checked in 2023019 and 2 other sigs
2021-12-12T07:04:15 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'et.DocVBAProject' is checked but not set. Checked in 2020170 and 0 other sigs
2021-12-12T07:03:48 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-user_agents.rules:250 uses unknown classtype: "pup-activity", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:44 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-malware.rules:15756 uses unknown classtype: "credential-theft", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:41 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-malware.rules:9890 uses unknown classtype: "social-engineering", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:41 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-malware.rules:8962 uses unknown classtype: "external-ip-check", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:39 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-info.rules:694 uses unknown classtype: "coin-mining", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:37 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-exploit.rules:800 uses unknown classtype: "exploit-kit", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:37 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-dns.rules:112 uses unknown classtype: "targeted-activity", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:37 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-attack_response.rules:488 uses unknown classtype: "domain-c2", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:03:37 suricata[4057] [100200] <Warning> -- [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - signature at /usr/local/etc/suricata/opnsense.rules/emerging-activex.rules:788 uses unknown classtype: "command-and-control", using default priority 3. This message won't be shown again for this classtype
2021-12-12T07:02:25 suricata[4057] [100200] <Notice> -- rule reload starting
2021-12-11T13:44:20 suricata[4057] [100200] <Notice> -- all 5 packet processing threads, 4 management threads initialized, engine started.
This might be a misconfiguration from my side but I cannot see where. All rules are updated and enabled.
Not sure what I have to modify to have Suricata working... Disabling it is a workaround, but not ideal.
I was checking similar issues and found some posts, but nothing useful...
21.7.7 is out. This will change testing outcomes, but I'm locking suricata, and upgrading to 21.7.7.
I'll test this way for a few days, and if there's no issues, I might unlock suricata, and wait a few more days.
Unfortunately after a day or so on 21.7.7 (keeping suricata from 21.7.5) I started having issues with anything voice related (Microsoft Teams for example), but no network cutout like before. Restarted everything, same issue right away. Unfortunately I don't have the cycles to look into this. Reverted to 21.7.5, and all seems fine again, but could also simply be coincidence. Sorry for not having time to spend on this and gather useful details.
In my case, keeping 21.7.6 with Suricata off (it was stopped automatically, without my intervention, probably something went wrong) I have not experienced the issue again in the past days. I think that for some reason some of the services were probably shut down automatically and that is why I experienced the DHCP/DNS problems.
I am still not sure how to research the source of the problem, but at least it is not repeating itself.
Start with basics and make no assumptions like version Y is broken and going to version X everything is fine. It'll take you nowhere near finding a solution. That is because you don't know what the problem is, which in fairness to you, is what you said from the start. It might be another problem unrelated to yours in the first place.
QuoteDuring the last two days I went to bed with Internet access and I woke up without it
Break the problem down. Take a client you can interact with like a pc. Does it have name resolution working? Has it got an ip, does it appear in the firewall traffic when you attempt to get through it?
You need to be ready to interact with this client and OPN at the same time, hence a PC is good.
QuoteWhen Windows tries to fix the issue by reseting DHCP, I lost the assigned IP (DHCP not working?)
Maybe I misunderstand but that is what is meant to happen. It says, hey dhcpd server, I want a new ip please, which one should that be? Gets one. Unless it has been reserved at the dhcp server, it might get the same or another one.
Finally, describe your setup. So many variables.
After many more days, I can confirm that the problem has not been reproduced again.
So I was not able to see the problem again or to find ways to research the issue. I think it was something making DHCP and DNS services go down.
To be clear (especially wrt the previous post), you are referring to the other person, I never said this was a version problem. I did not know what the problem source was but
I DO know it started after I rebooted the router in a long time after a version update --
BUT this does not mean that it is a problem of the new release, I do not know. I was just looking for ways to research what the problem was.
This:
os-dyndns (misconfigured) 1.27_1 173KiB OPNsense Dynamic DNS Supportis solved via:
https://forum.opnsense.org/index.php?topic=22529.0#msg111227
Quote"System: Firmware: Status", bottom right: "Resolve plugin conflicts" option "Reset all local conflicts"
The only thing that stays is a Suricata problem, but I think this should be reported in a new forum post.
I'm glad the that issue is resolved for you! :)
For the "version" discussion; whether it's a configuration that no longer jives with an updated package, or a package that is "broken", old definitions causing issues with Suricata, or any other case, coming only after an update and can be repeated, this is a versioning issue. This does not mean that a package is broken per se, but a set of condition lead to an issue with some package version.
With verasense no longer experiencing the problem, I'll take a deeper look time permitting.
I updated to 2.7.7 with suricata 6.0.4, and the network drop issue reappeared right away. Restarted the OpnSense VM, same issue.
I then disabled suricata, and no issues at all. I then re-enabled IPS (suricata), and no issues, aside from wifi calling and Teams not letting me talk, but I could hear just fine. Looking into it, a rule "Conficker-C P2P encrypted traffic UDP Ping Packet" was blocking outbound UDP traffic on port 4500; setting this to "Alert" only resolved this issue. This last issue is not directly related to this post, but seems to support the suggestion that something changed in the behavior of suricata, as I had not added any rules and had updated to the latest rulesets in all cases. This rule had been configured the same way in all cases, and only started to cause issues after moving to suricata 6.0.4.
I'm not a fan of the "solution", as disabling and re-enabling a service does not resolve the actual problem, akin to a duct tape solution, but I am not in a position to look under the hood and see if a caching issue, configuration change between version, etc... could be the cause.
All seems good now on my end.
https://en.wikipedia.org/wiki/Conficker
Conficker D 2009-03-04
HTTP pull
Downloads daily from any 500 of 50,000 pseudorandom domains over 110 TLDs
P2P push/pull
Uses custom protocol to scan for infected peers via UDP, then transfer via TCP
Blocks certain DNS lookups
Does an in-memory patch of DNSAPI.DLL to block lookups of anti-malware related web sites
Disables Safe Mode
Disables AutoUpdate
Kills anti-malware
Scans for and terminates processes with names of anti-malware, patch or diagnostic utilities at one-second intervals
Downloads and installs Conficker E