"Network is down" error on liveboot/fresh install

[coatmaker618]

I had an OPNsense install that was working until I accidentally uploaded a bad config and did a reset instead of a restore (that was the day I learned a reset deletes your local backups, whoops!).  As I tried to recover I ran into a weird problem where OPNsense states that my interface is down & I cannot ping anything from it (see attached photo) or ping it or access the webgui -- which is the current problem that I am struggling with.

I have tried a factory reset, even going so far as to try and boot from the live boot media.  Unfortunately I always get the same "ping: sendto: Network is down" error.

I was suspecting it was a driver issue and tried booting from the live image. I tried the most recent (as of today) and it did the exact same thing. When I tried the live image I originally used for the previously functional install (hoping that rolling back might fix the driver issue), I fatfingered the the interfaces (getting them backwards) and got a VERY interesting result -- the interface that was previously down is now getting DHCP just by being called a WAN instead of LAN.  What is going on here, I'm totally at a loss! And how do I get past the "network is down" error to a usable interface/network?

Please let me know if there's any other information that would be helpful for me to provide about the system to get it working again!

Notes:
1. I have photos (literally photos of the screen since it's all terminal) but cannot get them small enough to post, any suggestions are welcome if thhey would be helpful in troubleshooting.
2. This router is not hooked up to the actual internet (as the interfaces are being problematic), so security concerns while I'm testing are minimal.
3. I am using a NIC for these interfaces, but they do show in the list of interfaces so I assume that means the drivers are correct?

[coatmaker618]

Just to confound the issue more:

If I use the liveUSB (running as root instead of installer) the network seems to work -- I get a DHCP lease from the current router & whether using a static IP or DHCP, I can ping the current router. However when I install OPNSense using that liveUSB it all falls apart as I get the "network is down" error when I try to ping the current router & no DHCP lease :(

[Seimus]

Did you try to format the disk before a fresh installation?

Regards,
S.

[coatmaker618]

Did you try to format the disk before a fresh installation?

Regards,
S.

Yup, I've tried to reinstall (including formatting) several times.

[Seimus]

What kind of HW do you have (CPU, NICs)?
When you say interface is down do you mean LAN, WAN or both?
Are you able to connect to the GUI at all to check if that interface is enabled?
What does the output of these commands saying?

Code Select Expand

ipconfig
ipconfig -l
Regards,
S.

[coatmaker618]

What kind of HW do you have (CPU, NICs)?
When you say interface is down do you mean LAN, WAN or both?
Are you able to connect to the GUI at all to check if that interface is enabled?
What does the output of these commands saying?

Code Select Expand

ipconfig
ipconfig -l
Regards,
S.

Sorry for the delayed response, but I believe it was multiple issues combined together.

Just for completion, the answers to your questions were that I'm using a MINISFORUM MS-01 (which uses "12th Gen Intel(R) Core(TM) i9-12900H (14 cores, 20 threads)" per the OPNSense webgui) with a Mellanox (read: Nvidia) 2x 25 Gbe NIC. And ifconfig was showing all 6 network interfaces as existing but only sometimes showing an IP on the LAN & sometimes not.

I don't think I emphasized it, but this system not yet `in production` so the WAN is disconnected except when I was using it for testing.

Anyway, there ended up being 3 problems -- only 1 of which is at all related to OPNSense.

1. NIC was overheating. Turns out 25 Gbe NIC run hot, and putting them in a small box doesn't help. (I did however find that BSD/OPNSense has a package called `freeipmi` which reports to be able to read the NIC temp. This should be a separate post.)
2. This system uses vPro, which shares a network interface with the OS. vPro has an option to isolate the interface but OPNSense was reporting it anyway -- I suspect (but cannot confirm) that despite the interface being visible vPro was probably mucking with the traffic.
3. This is the only 'real' OPNSense issue, but there seems to be some default firewall rule blocking all network traffic to the GUI (I did change it from 443 to 8443) on all interfaces (LAN & OPT1).  While I won't claim the system is stock, I have NOT gotten to the point where I've touched the firewall rules, so I don't know why I need to disable the service to load the page.  I assume it's something dumb that I'm overlooking, but that's a tomorrow problem!  For now, just excited I got (most of) the original problem solved.

[Patrick M. Hausen]

Did you set a default gateway on LAN?

[coatmaker618]

I did! Seemed the easiest way to get updates & packages from the internet.

Does adding a gateway modify the firewall?

[Patrick M. Hausen]

Does adding a gateway modify the firewall?

Yes. Any interface with a gateway is considered a WAN interface ("LAN", "WAN" and friends are just names)
and any reply packet on such an interface is by default forced to go to the gateway instead of the originating
host.

Firewall > Settings > Advanced > Disable force gateway

to switch that off.

[coatmaker618]

Does adding a gateway modify the firewall?

Yes. Any interface with a gateway is considered a WAN interface ("LAN", "WAN" and friends are just names)
and any reply packet on such an interface is by default forced to go to the gateway instead of the originating
host.

Firewall > Settings > Advanced > Disable force gateway

to switch that off.

Disable force gateway is now checked. However there still seems to be something both preventing me from removing gateways (I can disable but when I try to delete them they just come back fully enabled) as well as create a WAN gateway (when I create an IPv4 gateway on the WAN the priority automatically becomes defunct instead of whatever is set, IPv6 on WAN is currently active & default).

Edit: Fixed the gateways on OPT1 & LAN, they were due to me setting them to DHCP (easy way to test if the network is working, or so I thought). Anyway, those are gone now, down to only gateways on WAN (IPv4 & IPv6).

That said, the IPv4 gateway (on the WAN) still says Priority is defunct.

[coatmaker618]

TL;DR Sitrep: I'm so close I can feel it! I have everything working except DHCP assignments (I'm using Kea), using manual/static IP everything works. Also the webgui widget only shows IPv6 gateway but the System: Gateways: Configuration page shows both IPv4 & IPv6.


Full Sitrep:


It turns out that plugging in a cable to the WAN fixed the issues with the gateway reporting defunct (I was avoiding that as I wanted to finish configuring the system before I connected it to the internet). There were reboots involved as well, which may have factored in.

I currently have LAN & VLANs on OPT & WAN working in the sense that they are up and report up and can ping things on them.

However there is one quirk remaining: The DHCP server (Kea, as it's recommended per https://docs.opnsense.org/manual/isc.html) on this OPNSense system does not seem to be issuing leases on the LAN. OPNSense has a static IP of course, but the desktop I am using to debug does not get an IP from Kea.

Please note that I am new to Kea, as I was using Isc before. That said, Kea looks to be running (it's shown in OPNSense services & the service is enabled) & is assigned to all non WAN interfaces (LAN & all VLANs, but not WAN). I added all the subnets and created individual reservations for each IP.  That said, when i go to the Kea leases page I see no leases issued -- so presumably it's not happy.

Weirdly, I don't see an option to enable ISC on the LAN interface as that interface is not listed.

All this means I am not being issued an IP address via DHCP but if I statically assign an IP I can access the webgui on OPNSense fine. If I assign a DNS server I can access the internet (that's how I'm writing this now).

I am still concerned about the gateway as I only see the IPv6 on the main page widget, but when I open the proper page (System: Gateways: Configuration) I see IPv4 & IPv6.  They both have priority 254, are the only gateways and are both listed as active.  So I'm assuming it's just an issue with the widget?  It also seems to work (given as static IP) as it can find the internet.

[coatmaker618]

Just a quick update, I got the widget to correctly display the gateway.  While restarting the system did not fix that issue, deleting and re-adding the widget did.  I suspect it was a cached list somewhere in the widget itself where the widget was querying only the gateways it knew rather than querying the

Still no luck on the DHCP server not providing IP addresses.