Disconnects about an hour after update/boot

[Solaris17]

Current running system

OPNsense 17.7.r2-amd64
FreeBSD 11.0-RELEASE-p10
OpenSSL 1.0.2l 25 May 2017

Update available via builtin update

Code: [Select]

bsdinstaller 17.7.r_2 17.7 upgrade
ldns 1.7.0 1.7.0_1 upgrade
opnsense 17.7.r2 17.7 upgrade
opnsense-lang 17.7.r2 17.7 upgrade
opnsense-update 17.7.r1 17.7 upgrade
strongswan 5.5.2 5.5.3 upgrade
unbound 1.6.4 1.6.4_1 upgrade
 
My server runs in hyper-V on a dedicated NIC that is not transparent to the OS. After running the update to the above changes I lose WAN but not LAN. However it might be more than DNS I cannot ping direct IPs either. If I reboot the unit it works fine for around an hour or so then stops again.

I have since recovered the VM via a backup to the version in the first line and everything appears to be working correctly.

If I can be of any assistance please let me know.

[franco]

Hi Solaris,

The fix for "unable to update" is in the release notes for 17.7:

https://github.com/opnsense/changelog/blob/master/doc/17.7/17.7#L63-L64

I've posted a test patch in the other thread.


Cheers,
Franco

[Solaris17]

Hi franco, thanks.

I didnt have an issue updating that I saw. Only that I stopped being able to resolve IP/DNS after the upgrade.

Are these somehow related to the hotfix you posted?

[franco]

i saw 17.7.r2 and "update" infos, I posted the note, but apparently all is well.

You are on 17.7 now and it doesn't work? This was ok for 17.7.r2?

[Solaris17]

Hi Franco.

Yes when I have the current available updates listed in the OP when I "upgrade" to 17.7 I ru into issues after about an hour of uptime. restarting services does not seem to resolve the issue. I have to restart the VM. At which point it will work again for around an hour sometimes less.

I reverted to 17.7_r2 and all appears stable.

sorry for the trouble. I will provide any info I can for you. Just not sure how to diagnose it for you.

[franco]

The distance between 17.7.r2 and 17.7 is really tiny.

It can't be a core change, so the kernel must be it...

Can you boot cleanly after upgrade by using the boot menu booting the old kernel? I think it's option 4, but not sure.


Cheers,
Franco

[Solaris17]

Thanks for getting back to me Franco. I proceeded with the update then during the boot process (5) I selected kernel.old

Will get back to you and let you know how it goes.

EDIT::

It didnt work made it about 1hr 30min before the system stopped resolving even on kernel.old. I am not sure what to logs to pull for you but this router is production so I already recovered. I got this output from the standard log page. From inside the GUI.

Code: [Select]

Aug 6 12:15:14 configd.py: [d691441b-866a-410e-9aa2-e645bbc41103] request vmstat interrupt counters
Aug 6 12:15:14 configd.py: [015d979b-23a3-44bb-9fc2-de3f93ace315] request pfctl byte/packet counters
Aug 6 12:15:14 configd.py: [1eb46c27-9e08-4238-9a23-430eb88c8dce] request mac table
Aug 6 12:14:22 configd.py: [1d9500ae-9006-4062-be7d-2e2e974c9949] get suricata daemon status
Aug 6 12:14:20 configd.py: [a4712e43-85ec-4c0f-be43-59596b4b099d] request pfctl byte/packet counters
Aug 6 12:14:15 configd.py: [5aa3aa33-78ae-4d8b-bd7f-7189320db26a] request pfctl byte/packet counters
Aug 6 12:14:14 opnsense: /index.php: Successful login for user 'Solaris17' from: 10.0.65.134
Aug 6 12:01:05 sshd[4624]: Disconnected from authenticating user root 10.0.0.6 port 3737 [preauth]
Aug 6 12:01:05 sshd[4624]: Received disconnect from 10.0.0.6 port 3737:11: Session closed [preauth]
Aug 6 12:01:04 sshd[4548]: Disconnected from 10.0.0.6 port 3736 [preauth]
Aug 6 12:01:04 sshd[4548]: Received disconnect from 10.0.0.6 port 3736:11: Session closed [preauth]
Aug 6 12:00:00 sshd[85602]: Did not receive identification string from 10.0.0.6 port 3599
Aug 6 11:58:32 configd.py: [b3f4805b-3276-471d-aaac-8d9634357c4b] request pfctl byte/packet counters
Aug 6 11:56:32 configd.py: [52e592d3-eae8-4116-9519-d6103b98b3c1] request pfctl byte/packet counters
Aug 6 11:53:10 configd.py: [ba78d3d7-e73c-4a13-9ea0-3fd17be446c4] request pfctl byte/packet counters
Aug 6 11:51:46 configd.py: [e7e89829-278a-4ed6-baeb-89a3d25af3d7] request pfctl byte/packet counters
Aug 6 11:49:25 configd.py: [65d45237-4b7d-4eba-b932-60a7f8f30fc5] request pfctl byte/packet counters
Aug 6 11:46:40 configd.py: [45c06614-8869-4583-96d8-974819e2fe12] request pfctl byte/packet counters
Aug 6 11:44:12 configd.py: [694b0891-a034-4fff-98a8-1acdcd56826a] request pfctl byte/packet counters
Aug 6 11:40:56 configd.py: [c4be2f1a-a0ac-4e8f-a751-068af6254a6e] request pfctl byte/packet counters
Aug 6 11:40:48 configd.py: [4e96dd15-4bf1-4ce9-b9b9-708c270b3c67] request pfctl byte/packet counters
Aug 6 11:40:42 configd.py: [60148f92-1103-4e9f-828d-22ec8eb72c77] request pfctl byte/packet counters
Aug 6 11:40:36 configd.py: [f3e151fd-7ff5-4bce-a720-d52d2caa9f69] request pfctl byte/packet counters
Aug 6 11:40:30 configd.py: [fbb0375d-b15e-4a60-97b4-f9e469fe2325] request pfctl byte/packet counters
Aug 6 11:40:24 configd.py: [fb2cd39d-7202-4d0a-a05a-63ca133856d1] request pfctl byte/packet counters
Aug 6 11:40:18 configd.py: [0c88ff7b-4e8a-4db2-86ab-82a43740e9a1] request pfctl byte/packet counters
Aug 6 11:40:12 configd.py: [905ffee6-a4ab-4ee1-b0df-512c7e7fb7bc] request pfctl byte/packet counters
Aug 6 11:40:06 configd.py: [5c15acd9-1764-4455-a8c1-2363338b3bc5] request pfctl byte/packet counters
Aug 6 11:40:00 configd.py: [fdda5c3e-daf8-4424-9624-c2d309637384] request pfctl byte/packet counters
Aug 6 11:39:54 configd.py: [a5b1dcdc-b5e5-4d13-a219-022794d86fcd] request pfctl byte/packet counters
Aug 6 11:39:48 configd.py: [5a87e747-6bfd-45d1-b086-64b5d464012d] request pfctl byte/packet counters
Aug 6 11:39:42 configd.py: [68300a72-e492-4541-81ee-c1e98fc2cc79] request pfctl byte/packet counters
Aug 6 11:39:36 configd.py: [18ca8332-7ada-47c0-b8bd-5ed663b88b40] request pfctl byte/packet counters
Aug 6 11:39:30 configd.py: [089be530-b5ae-47b9-96f5-817eb3f630f3] request pfctl byte/packet counters
Aug 6 11:39:24 configd.py: [b1f31921-a7f9-4003-b279-1235fa1e3683] request pfctl byte/packet counters
Aug 6 11:39:18 configd.py: [8897996f-ef2a-4f71-a03a-549631a5b1c4] request pfctl byte/packet counters
Aug 6 11:39:12 configd.py: [e63ec74d-9911-4630-94b3-2c4f57bd9b1e] request pfctl byte/packet counters
Aug 6 11:39:06 configd.py: [2f54aebd-8979-4e4e-9737-30a0e8958cc1] request pfctl byte/packet counters
Aug 6 11:39:00 configd.py: [76063869-cc10-4a91-8d4a-ce984fac77d5] request pfctl byte/packet counters
Aug 6 11:38:54 configd.py: [741a657a-ef8c-4d24-b4b2-3b3102510336] request pfctl byte/packet counters
Aug 6 11:38:48 configd.py: [8f99338d-7bca-4fa6-8dfe-03e8b4e45350] request pfctl byte/packet counters
Aug 6 11:38:42 configd.py: [27ac9619-d9b8-4dac-9359-b55d97a144b6] request pfctl byte/packet counters
Aug 6 11:38:36 configd.py: [54a93444-38c1-4fd2-9088-3e05a9fed05f] request pfctl byte/packet counters
Aug 6 11:38:30 configd.py: [8401df78-f3ed-4ca3-a5b4-f8a96233eaa0] request pfctl byte/packet counters
Aug 6 11:38:24 configd.py: [59e6f43d-73b2-41b2-98a2-17d5d0098d4f] request pfctl byte/packet counters
Aug 6 11:38:18 configd.py: [4c59a1ac-e8a3-41cc-a536-d19b2cf00a36] request pfctl byte/packet counters
Aug 6 11:38:12 configd.py: [7b5c9320-1d39-4a45-8e08-611c617a2dc4] request pfctl byte/packet counters
Aug 6 11:38:06 configd.py: [e90829fd-624e-4fb6-af13-e7c0fd0fd8c7] request pfctl byte/packet counters
Aug 6 11:38:00 configd.py: [cfdadc59-f945-4752-beb5-34ca71352dce] request pfctl byte/packet counters
Aug 6 11:37:54 configd.py: [d22e3da8-097c-4769-9efc-ec78eeeae6f5] request pfctl byte/packet counters

[franco]

This all seems to be leading nowhere in particular, which is both good and bad at the same time.

In the log there is a line about Suricata. Do you have IDS+IPS enabled? With Hyperscan? We're tracking down an issue there which seems to only be fixed by reverting to Suricata 3.2.2:

# pkg install -f https://pkg.opnsense.org/snapshots/suricata-3.2.2.txz

(don't forget to stop + start the service)


Cheers,
Franco

[Solaris17]

I do use Suricata (IDS+IPS), no lines about it however. I use it via default and not hyperscan though.

[franco]

Ok, thanks for the info. Try to disable or use the older version. This should really help.

[Solaris17]

It may be more worth it to run the old one then to disable it completely fo bug reporting so I will update and run the old version.

How do you want me to add to repo?

EDIT:: As of now I will simply run with it disabled. When you get back to me on how to add the repo for the older version I will then down grade suricata and re-enable. kill 2 birds.

EDIT 2:: Its been 2 hours with Suricata disabled after the upgrade and I have not been disconnected once. Current build in dash is

Code: [Select]

OPNsense 17.7-amd64
FreeBSD 11.0-RELEASE-p11
OpenSSL 1.0.2l 25 May 2017


[franco]

Ok, so that is it then. I already posted the command to install the older Suricata:

# pkg install -f https://pkg.opnsense.org/snapshots/suricata-3.2.2.txz

You can lock it if you want (also works from System: Firmware: Packages):

# pkg lock suricata

Locked packages will not be upgraded (except for in major updates).

Marking this as a workaround, we're investigating where this came from, but it's good that we can rule out Hyperscan as a potential culprit.


Cheers,
Franco