No carrier after install

[JavaMan07]

I'm new to OPNsense, and struggling to get it routinely functional.
When I run it in live mode it works great, stable even overnight. Since I'm working to get it installed, I haven't left it running longer than that yet. In live mode, it does not auto assign the interfaces right. I have to eiher assign the ue0 interface to WAN using the UI, or ssh root login. Either method seems to work fine while it's running in live mode.

When I have a working setup, I login via SSH to installer@192.168.1.1 to complete the install to SSD, then reboot. After that I can't get it to work, the WAN interface keeps showing "no carrier" and won't get an IP address. At this point, I've tried installing three different times, some part of the working live-mode setup is not getting pushing into the installed system. I've even tried backing up all system settings (from the live-mode) and restoring those settings when it's running the installed version, but that still won't activate the WAN side.

I am fairly confident that the cables are good, as when it's in live-mode it works great. I even loaded i by doing speed tests or large downloads, while I wiggled and pulled on the ethernet and USB cables, with no hiccups in transfer speed. I figured any minor problem with the cable is more likely to show up during high network traffic versus just sitting there.


Hardware: Lenovo ThinkCentre M600 Type 10G9. LAN is the internal gig ethernet. WAN is a Wavlink USB to gig ethernet adapter, which works fine on other PCs. Within the UI it identifies as I do plan on replacing the USB adapter with a M.2 key A or key E ethernet adapter, and probably will get a 2.5Gb version.

Live version info  OPNsense 22.7-amd64, FreeBSD 13.1-RELEASE, OpenSSL 1.1.1q 5 Jul 2022

Here's the ifconfig for the ue0 interface when it's running in live-mode
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=80088<VLAN_MTU,VLAN_HWCSUM,LINKSTATE>
        ether 80:3f:5d:06:8c:dc
        inet6 fe80::823f:5dff:fe06:8cdc%ue0 prefixlen 64 scopeid 0x7
        inet 192.168.0.14 netmask 0xffffff00 broadcast 192.168.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>


Here's the ifconfig for the ue0 interface when it's running as installed to SSD
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=68009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 80:3f:5d:06:8c:dc
        media: Ethernet autoselect (none <half-duplex>)
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>



Here's the usbconfig output
root@OPNsense:~ # usbconfig
ugen0.1: <Intel XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen0.2: <General UDisk> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (100mA)
ugen0.3: <GenesysLogic USB2.0 Hub> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (100mA)
ugen0.4: <vendor 0x0424 product 0x2514> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (2mA)
ugen0.5: <vendor 0x8087 product 0x0a2a> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
ugen0.6: <GenesysLogic USB3.1 Hub> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen0.7: <Realtek USB 10/100/1000 LAN> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (72mA)

[cookiemonster]

inet 192.168.0.14 netmask 0xffffff00 broadcast 192.168.0.255
That is a private IP, so it suggests there's another device in front of OPN getting the WAN address when you're in live mode testing.
Not necessarily wrong but curious.
I'd check with dmesg what happens with that interface (WAN) during boot.

[JavaMan07]

That is correct, my ISP modem/router combo unit. I can only get internal IP addresses from it, even when I place something in the DMZ.
I can see on my desktop that I sometimes get port scanned from various IPs outside the US. So I don't really trust the "advanced cyber security" provided by the ISP router. I'm putting the OPNsense between the ISP modem/router and my internal network (NAS, Proxmox, workstations)

[WN1X]

Replace your modem/router with a standalone modem. This will eliminate the double NAT situation you have with running OPNSense (or any other firewall for that matter).

[cookiemonster]

It doesn't explain why the behaviour changes from live to installed but yes, it would help and preferable when not fully conversant with the networking ways.
[quote ]
That is correct, my ISP modem/router combo unit. I can only get internal IP addresses from it, even when I place something in the DMZ.
I can see on my desktop that I sometimes get port scanned from various IPs outside the US. So I don't really trust the "advanced cyber security" provided by the ISP router. I'm putting the OPNsense between the ISP modem/router and my internal network (NAS, Proxmox, workstations)
[/quote]
That is the expected behaviour. Your current modem/router combo is terminating the line, dialing up and getting the WAN ip. A machine on the DMZ zone won't be doing another request for a public ip unless setup to do so, and would only get one if your isp has assigned multiple public ips on your service -something you had to pay for and be on a business or specialised prosumer package-. Maybe you do have it but you haven't stated it.

For a simple check, see if your modem/router can be put in passthrough ie. bridge mode. Then it will only get the wan.
Alternatively, take the modem/router out of the chain i.e. power down and disconnect, and connect only OPN and go through the installation routine so it gets a clean chance to detect the WAN.

[JavaMan07]

To get a different modem, without a built-in router, I would have to get the business plan. They do not sell modem-only units to home users, gave me some BS about providing better security this way.
To get a second external IP address, I would have to buy the business modem and get the business plan. Depending on which customer support person I talk to, they may require a copy of a business license to get the business plan. Anyway, the business plan would raise my monthly cost from $65 to $245, not work it for me. A business plan is not faster either, I'm limited by their DSL technology.
The home modem/router does not have a bridge mode, their support tech could not understand why anyone would want an insecure connection. So far the best I can do to eliminate their stuff is to turn off the WiFi of their modem/router unit, and put my own solution behind it (along with the double NAT issues that causes).

[WN1X]

If this were my ISP, I would be looking for a new provider.

That said, who is your ISP? They may not sell a standalone DSL modem, but there are a number of them available for sale from companies like NetGear and TP-Link.

[cookiemonster]

To get a different modem, without a built-in router, I would have to get the business plan. They do not sell modem-only units to home users, gave me some BS about providing better security this way.
To get a second external IP address, I would have to buy the business modem and get the business plan. Depending on which customer support person I talk to, they may require a copy of a business license to get the business plan. Anyway, the business plan would raise my monthly cost from $65 to $245, not work it for me. A business plan is not faster either, I'm limited by their DSL technology.
The home modem/router does not have a bridge mode, their support tech could not understand why anyone would want an insecure connection. So far the best I can do to eliminate their stuff is to turn off the WiFi of their modem/router unit, and put my own solution behind it (along with the double NAT issues that causes).

I wasn't suggesting to attempt going for that, just because you wrote "I can only get internal IP addresses from it, even when I place something in the DMZ." so I assumed you wanted to say host your own webserver on a different public IP to your "main one".
I understand the pickle with Isp though. Same in the UK, some IPS provide a router that their custom UI doesn't allow bridge mode. The solution here is to replace it completely with any other that does. BUT that works because it's easy to find the required authentication details needed to enter in the replacement unit for PPoE, or by the identification of the plant i.e. the IDs of physical cable.
Anyhow, if you can get those, easy solution: replace just the modem part, put in bridge, OPN after.

In the meantime we're digressing a little. Post what dmesg shows, we're looking for clues what might be the problem despite the double nat.

[JavaMan07]

@WN1X  This is with CenturyLink (DSL provider), my other option for wired internet is with Xfinity (the cable provider). There are no other wired providers in this neighborhood.
I looked at some WISP wireless providers, but I'd be dropping from 110Mbps/10Mbps to like 15Mbps/2Mbps or less upload. Unfortunately, the slow speeds available via WISP providers are not feasible for me.

@cookiemonster  I was not able to get to dmesg to collect logs. This problem only occurs after I complete the install and reboot, which it apparently disables SSH access. Since there are all the warnings about how bad enabling SSH is, I left it off.

I was finally able to get it to work, after rebooting a number of times it finally just worked. I then updated the system, which required a reboot. It still worked and continued working after some additional reboots.

Hopefully, I'm only stuck here with this ISP for a few more weeks. We've been renting for a year while our house is under construction. We're supposed to close on Sep 7th, we're just waiting to hear from underwriting to give final approval. After we move we'll finally have fiber internet. I'm so looking forward to it. It's from a small local company, so I'm sure it will be much better service than these large *$& ISPs.

[WN1X]

Frankly, DSL sucks. Your internet access will be much better once you move and get fiber. I'm spoiled with gig fiber the past two years.

[JavaMan07]

While DSL and cable are inferior technologies for broadband internet, it's still the company running the service and support that is the root of the problem.
I used to have cable internet. During the busy time of day, 3pm to 9pm, service would really slow down. My 100Mbps/1Mpbs would slow down to 5Mbps/0.05Mbps which would pretty much make the internet useless. After complaining multiple times, paying for them to come out and test my line (during good hours) and say the problem is my internal apartment wiring. My only alternative wired ISP was a DSL provider, so I decided to increase my subscription rate. My 500Mbps/5Mpbs would slow down to 65Mbps/1Mbps which would kill video conferencing but still allow for functional internet and streaming.

With DSL now I don't get any slow down based on time of day, just the random (maybe once a month) issues where some websites break or get very slow. But my 100Mbps/10Mbps is the fastest speed they offer. The faster upload speed is a definite plus compared to cable, at least for work from home and video conferencing.

What I find ironic, is the big providers stick to cable/DSL because they say fiber is too expensive to lay, too expensive to maintain, etc. While the small providers that specialize in fiber internet charge lower prices, are profitable and growing. Albeit slowly since they can only grow in new housing developments. No one seems to be interested in building in established neighborhoods.

[Vesalius]

What I find ironic, is the big providers stick to cable/DSL because they say fiber is too expensive to lay, too expensive to maintain, etc. While the small providers that specialize in fiber internet charge lower prices, are profitable and growing. Albeit slowly since they can only grow in new housing developments. No one seems to be interested in building in established neighborhoods.

Established companies have more regulations and potential blow back for putting fiber in only in profitable new neighborhoods and skipping over the ones they already service. New companies don't suffer any local/federal/political/contractual ramifications for actively expanding into those same areas because they don't have the "baggage" of preexisting older neighborhood customers.

[JavaMan07]

Well, at least in neighborhoods built since 2016, the residents have three providers instead of just the two (Xfinity for cable, CenturyLink for DSL, and various fiber providers in regions of the city).

I thought my problem with the WAN link was solved, but after doing a restart today, its back. I collected the dmesg output, and attached it here. After a few reboots trying to resolve the WAN issue, I cannot connect to it at all now. Each time I reboot I just get "no route to host"

[cookiemonster]

I got the feeling from the ups and downs of the usb network interface, that it is at least part of the problem.
From your signature, the system seems to have a multiport Intel-based I350-T4 Nic.
I would try setting up one port for WAN, one for LAN, leave USB nic out completely and test it that way.

[JavaMan07]

It takes a bit of unplugging and plugging the USB back in to get it working, 2-7 times, but once it is working it works great till the system gets rebooted again. I'm glad that it's solid when it is working, but rebooting or unplugging things repeatedly till it works again is not something that the wife would be happy about.

I only have one ethernet port on this system. If it supports another natively, it's via some add-in card that I do not have.

My M.2 E key to ethernet adapter came in a few days ago. I was unable to use it yet because of the clearance between it and the 2.5" SATA tray. I should have my M.2 SATA drive tomorrow, so I'll be migrating to it and getting that ethernet adapter installed soon. Hopefully, that gets this working solidly