Messages

It is my experience that once a disk is failing it is best to first make sure to get all the information from the system, while it is still running. A system that is still runnning with disk issues, often does not reboot because of these disk issues. Next to that without my opnsense server I would not be connected to the internet (makes it hard to search google, download opnsense ISO's etc.). so I stalled addressing this issue for as long as I could (was working from home and although things appeared messed up , I still had internet)

Unfortunately the system started degrading over the days and finally my internet connection broke, dns and dhcp stopped working  (but opnsense would still be responding to ping). So I had everything prepared and I was ready to re-install the system but as I no longer had anything to lose, I just rebooted the system.

This fixed the issue, after reboot the disk showed up again:

Code Select Expand

root@OPNsense:~ # geom disk list
Geom name: nvd0
Providers:
1. Name: nvd0
   Mediasize: 128035676160 (119G)
   Sectorsize: 512
   Mode: r3w3e8
   descr: Hoodisk SSD
   lunid: 6479a747c030213d
   ident: M1YLCKC21272049
   rotationrate: 0
   fwsectors: 0
   fwheads: 0

root@OPNsense:~ #

And the SMART plugin does again show useable information:



So I think the title of this post should have been:
"System stopped working during upgrade to 24.1.9"

Apparently as no one replied to this post, I am the only one to experience this?

I am running:
OPNsense 24.1.8-amd64 on an Intel(R) Celeron(R) N5105 @ 2.00GHz (4 cores, 4 threads).

Using the webgui I tried to uprgade from 24.1.8 to 24.1.9 but the upgrade failed.
Now when I retry to do an update it shows:

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.1.8 at Thu Jun 27 11:00:38 CEST 2024
Fetching changelog information, please wait... done
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 845 packages processed.
Updating mimugmail repository catalogue...
Waiting for another process to update repository mimugmail
All repositories are up to date.
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking for upgrades (42 candidates): .......... done
Processing candidates (42 candidates): .......... done
Checking integrity...Child process pid=81326 terminated abnormally: Bus error
***DONE***

Logging into the console and trying from command line, I get this error:

Code Select Expand

Enter an option: 12

Fetching change log information, please wait... done

This will automatically fetch all available updates and apply them.

Proceed with this action? [y/N]: y

/usr/local/opnsense/scripts/shell/firmware.sh: less: Input/output error


Using a shell I get the same error when using "less":

Code Select Expand

root@OPNsense:~ # which uptime
/usr/bin/uptime
root@OPNsense:~ # which less
/usr/bin/less
root@OPNsense:~ # less
/usr/bin/less: Input/output error.
root@OPNsense:~ # uptime
11:04AM  up 15 days, 10:24, 2 users, load averages: 0.29, 0.31, 0.26


I am afraid my disk is failing so wanted to look at smartctl, but no device is shown in the webgui:


On the command line I have now idea which device to use:

Code Select Expand

root@OPNsense:~ # geom disk list
root@OPNsense:~ #


dmesg does show messages like this:

Code Select Expand

g_vfs_done():gpt/rootfs[WRITE(offset=114130190336, length=262144)]error = 6
g_vfs_done():gpt/rootfs[WRITE(offset=114156208128, length=32768)]error = 6
g_vfs_done():gpt/rootfs[WRITE(offset=114156470272, length=294912)]error = 6
g_vfs_done():gpt/rootfs[WRITE(offset=114156994560, length=294912)]error = 6
vm_fault: pager read error, pid 550 (ruby31)
vm_fault: pager read error, pid 4922 (ruby31)
vm_fault: pager read error, pid 11023 (ruby31)
vm_fault: pager read error, pid 21105 (ruby31)
vm_fault: pager read error, pid 26145 (ruby31)


I admit I am a total n00b when it comes to Freebsd , so is my disk indeed dying(dead?)

[Ip address]

Since these are a VM, did you have a look at the VM host and see if by any chance it also had the same IP address?
Just wondering if an address is handled out before that the virtual MAC is supplied. Do you have a physical computer you can connect and see if it gets the same 3.130 address?

Must admit that this was also my thinking, but what contradicts this is the fact that the DHCP log shows the correct mac-addresses. But I  did some tests to see if that could be the case (see below).

The lease of ip-address 192.168.3.130 is shown in the lease list under the correct mac-address.

The 3 proxmox hosts in my cluser have statically assigned leases:


   
   
   
   

Ip address	Mac Address	Hostname
192.168.3.10	5c:ba:2c:20:0d:9e	pve.familie-dokter.lan
192.168.3.11	3c:a8:2a:a0:63:28	gen8.familie-dokter.lan
192.168.3.12	bc:ee:7b:26:5e:da	silverlaptop.familie-dokter.lan

I tried a real (not vm  host)  (as suggested): it receives a lease for 192.168.3.132 (so not 130).

So I performed a small test by adding a static lease for the VM macaddresses:
02:57:e4:9e:f8:2b (new-test1)  --> 192.168.3.81
02:af:92:38:c9:46 (new-test2)  --> 192.168.3.81

Both work exactly as expected, I think that means that the mac-addresses are usable and set at the correct time..

Have you changed anything on your server?

The only thing that I did iis upgrade opnsense each time a new version was available, and change the port of the opnsense web interface (due to some upgrade issues with HAproxy described elsewhere on this forum). I probably did something else to break DHCP if I am the only one with this issue  but can't remember if I did that and what I did  :(

Did you notice that the DHCP discover from debian-two mac address (ef:41) is tagged as debian-one host?

Not sure what you mean with that, but I have rebooted debian-one and debian-two hosts a couple of times to get a clean log. This means that at some point one of the 2 would probably have asked for " the ip-address I got last time".

I have created 2 new hosts, with completely different mac-addresses and this is the dhcp4 log if I boot both at the same time:

Code Select Expand

2024-03-31T11:45:31 Informational dhcpd DHCPACK on 192.168.3.130 to 02:af:92:38:c9:46 (new-test2) via igc1
2024-03-31T11:45:31 Informational dhcpd DHCPREQUEST for 192.168.3.130 (192.168.3.1) from 02:af:92:38:c9:46 (new-test2) via igc1
2024-03-31T11:45:31 Informational dhcpd DHCPOFFER on 192.168.3.130 to 02:af:92:38:c9:46 (new-test2) via igc1
2024-03-31T11:45:31 Informational dhcpd DHCPDISCOVER from 02:af:92:38:c9:46 (new-test1) via igc1
2024-03-31T11:45:28 Informational dhcpd DHCPACK on 192.168.3.130 to 02:57:e4:9e:f8:2b (new-test1) via igc1
2024-03-31T11:45:28 Informational dhcpd DHCPREQUEST for 192.168.3.130 (192.168.3.1) from 02:57:e4:9e:f8:2b (new-test1) via igc1
2024-03-31T11:45:28 Informational dhcpd DHCPOFFER on 192.168.3.130 to 02:57:e4:9e:f8:2b (new-test1) via igc1
2024-03-31T11:45:27 Informational dhcpd DHCPDISCOVER from 02:57:e4:9e:f8:2b via igc1

Thank you fortaking  time to post some suggestions.. As I need to fix this:
to be continued

The range of dynamic ip addresses is: 101 tot 200 (so 130 falls within that).

There is no static lease on 130:
https://pasteboard.co/C41JPcohy7sD.png

Just found out that it is even worse: every "new"  lease requested gets ip-address 192.168.3.130

I have 2 virtual machines running on my proxmox server, When started the same ip-address is handed out  from  ISC DHCP to both, although they do have different mac-addresses. Not sure when started failing  but this has worked without issues.

Below you can see that 192.168.3.130 is given to both debian-one and debian-two, after both rebooted:

Code Select Expand

2024-03-28T16:39:28 Informational dhcpd DHCPACK on 192.168.3.130 to bc:24:11:cd:ef:41 (debian-two) via igc1
2024-03-28T16:39:28 Informational dhcpd DHCPREQUEST for 192.168.3.130 (192.168.3.1) from bc:24:11:cd:ef:41 (debian-two) via igc1
2024-03-28T16:39:28 Informational dhcpd DHCPOFFER on 192.168.3.130 to bc:24:11:cd:ef:41 (debian-two) via igc1
2024-03-28T16:39:28 Informational dhcpd DHCPDISCOVER from bc:24:11:cd:ef:41 (debian-one) via igc1
2024-03-28T16:39:20 Informational dhcpd DHCPACK on 192.168.3.130 to bc:24:11:4a:b5:1e (debian-one) via igc1
2024-03-28T16:39:20 Informational dhcpd DHCPREQUEST for 192.168.3.130 (192.168.3.1) from bc:24:11:4a:b5:1e (debian-one) via igc1
2024-03-28T16:39:20 Informational dhcpd DHCPOFFER on 192.168.3.130 to bc:24:11:4a:b5:1e (debian-one) via igc1
2024-03-28T16:39:19 Informational dhcpd DHCPDISCOVER from bc:24:11:4a:b5:1e via igc1
2024-03-28T16:39:10 Informational dhcpd DHCPRELEASE of 192.168.3.130 from bc:24:11:cd:ef:41 (debian-two) via igc1 (found)
2024-03-28T16:39:02 Informational dhcpd DHCPRELEASE of 192.168.3.130 from bc:24:11:4a:b5:1e via igc1 (not found)

Anything I can do or check to fix this?

So the first workaround was to move the admin website to a different port than 443

That fixed it! Thank you!

My system is headless so I am reluctant to start over (or even downgrade) Can anyone please give me some ideas on where to look?

I was running 23.x just fine but have an issue after upgrading to 24.1 (and 24.1.1) the OPNsense webgui cannot be reached. Even when using the ip-address of the OPNsense LAN interface I cannot reach the webgui but end up on my public webserver.

I found out that when I stop ha_proxy (using a shell) the web interface is available. The moment I start ha_proxy, it becomes unavailable again. This setup had ran fine before the upgrade to 24.x

Any help / pointers/ questions for more information would be welcome, can't really use OPNsense at the moment.

Does Suricata work on VLANs now?

Your reply made me recheck what the documentation says about VLAN, so I had another look and  the documentation does state:
"Interfaces to protect.  | When in IPS mode, this need to be real interfaces supporting netmap. (when using VLAN's, enable IPS on the parent)"

So I removed vlan from interfaces and added icg3 and now with it keeps running even with IPS switched on. Not sure why it did not work the previous time I tried this but I am sure I just messed up.

Bump..

That's a pity.
Because I can of course read the original HAproxy documentation, but that documentation describes how to use HAproxy config  files and does not explain where to look in the opnsense UI.

I would be more than willing to create a pull request for updated documentation but that means that first I need to understand it myself :-)

In this page: https://docs.opnsense.org/manual/reverse_proxy.html, the documentation references Haproxy How-Tos: https://docs.opnsense.org/manual/how-tos/haproxy.html
But those How-Tos no longer exist.

This is also true for the web interface of the haproxy plugin, it refers to the same non-existent howtos.

This basically means that at the moment there is no opnsense haproxy documentation at all

I have posted a similar problem :
Intrusion Detection stops after 1 minute

Assumed my issue was related to "changing to VLAN"  but maybe it is related to what you are reporting?