New setup - WIP and need help

[lew-adv-sol]

Forgive the bullet points and numbering, it didnt copy the formatting correctly

Hey everyone,
I am relatively new to OPNsense and softrouting. I have used this as a chance to find the shortcoming in my knowledge and have wholeheartedly succeeded.
Cutting the family off from internet puts my life at risk so I am challenging myself to seek out a efficient solution for my soho setup. I work for a MSP and need to get better at the routing basics as well.I've waited over 3 weeks to ensure I've exhausted my resources and saved you all from any easy troubleshooting[size=78%]. [/size]I have just gotten such inconsistant results and outcomes that I need assistance for future risk management.

My issue:
Different approaches to setting up the OPNsense router have resulted in the LAN or WAN losing connection/DHCP functionality in different forms

Static factors:
⦁   Router device: optiplex micro 3070 i5 9th gen 16g memory 128g nvme. Running baremetal OPNsense 23.7 fresh factory defaults w/ updates ran, realtek driver plugin installed.
⦁   Modem:BGW-320 500 At&t fiber modem (default configuration for testing with IP scope of 192.168.253.0/24, Gateway 192.168.253.254, DHCP, and packet filtering enabled, netscan can confirm it is on the network)
⦁   Interfaces: onboard intel 1g nic for LAN and auto select 2.5g Realtek usb to rj45 for WAN(normally/successfully used 4 usb to rj45 w/ LAGG before. Then wan IP DHCP broke again. reverted for testing)tested on other hardware with successful network connection
⦁   cat6 tested
⦁   ISP modem has another bridged eero mesh device handing out addresses to the house. While the house enjoys internet, the OPNsense struggles.

Setup is onboard nic of the Optiplex is the default LAN, the 2.5g is always WAN(previously no issues when it got working.) LAN and WAN are tested using either my win11 laptop or mikrotik 2011. OPNsense WAN is connected to port 1/Eero connected to port 2 on modem.
As of this post the installation was freshly downloaded and installed using rufus with no issue. I have left every setting as default for this testing.

Generating my issue from default settings:
⦁   Changing IP scope on LAN such as 10.22.0.1/24 will cause LAN DHCP to break as in no ip assignment or connection is blocks at the OPNsense gateway for the win11 laptop directly connected to LAN. Through webGUI and on CLI

•   I release and renew IP, released and registered DNS
•    When this works I moved on the next steps of attempting to configure, such as WAN set to static in which it just breaks WAN. thats coming up.
•    I check my ipconfig details and IF I get an IP address I ping the 10.22.0.1 address which is normally successful if I got my IP address. I then ping the modem gateway of 192.168.253.254(after unchecking bogon and rfc1918 blocking) if this good then I tracert to 8.8.8.8.
•     If this works, great I have internet out the gate.....until WAN drops for no reason and never recovers until reset or an unknown number of reboots modem and opnsense in which it still later breaks.
•    IF it doesnt have internet out the gate I start the following:
•    If no Ip get assigned I unplug and replug the usb to rj45 connection on the laptop after 30 second wait. If no IP I move on, If I do...I wait for it break.
•    I attempt to set a static IP with the 10.22.0.2 subnet 255.255.255.0 DNS 10.22.0.1. This normally does nothing extra if it was already getting a DHCP assignment and would fail if the DHCP assignment had already failed. It still sometimes break the connection and not recover...othertimes it works....Im not sure as to why.
•    Here is where I test the connectivity on another network successfully using the rj45 jack on the eero or modem.
•    I test the connection with the  mikrotik router on LAN, its equally hit or miss. If the laptop was not able to get DHCP the mikrotik normally isnt an exception but the experience is the same no matter the device.
•    IF I do get an IP but not internet I usually can not get past the interface gateway on a ping to 8.8.8.8 with the result 10.22.0.1 destination host unreachable even when the laptop has the 10.22.0.10 address from DHCP.
•    At this point I have cleared the device details and focus on the OPNsense router
•    If DHCP lease shows my device with IP I move on, If not I restart the service this usually does nothing but make me feel better....not really.
•    I verify timezone attempting UTC and CDT. No impact on LAN but definitely was a pain point for WAN early on in this experience.
•    No sign of pings or any rejects/block in live view of firewall -still with defaults rules-  I will include firewall logs from the pings and attempts web access
•    I go throught the wizard again at this point, sometimes this is temporary fix as they all have been.
•    sometimes I revert back to 192.168.1.1 or something entirely different such 192.100.0.1 and they are equally hit or miss and just as reliable.
•    I attempted both IPv6 enabled and disabled.
•    Bogon was already unchecked but attempt alone and in combonation with the other attempts.
•    The end of the LAN war...at this point it has been reset to default and left working or miraculously the LAN IP scope change stuck the 1st or 15th time.   

-Side story - When it was working with LAGG and 5 vlans, the vlan interfaces would go down just as much as the LAN or WAN interface.

⦁   Changing WAN IP to a static such 192.168.253.1/24 from DHCP assignment will cause the WAN to repeatedly fail to get a gateway determined.Through webGUI and on CLI.  It will show on the ISP modem  when it does connect as the hostname @ 2500mbps and ready for internet...just no internet.

•    I can manually place the gateway at 192.168.253.254 as it located on its own before and it will still fail. when it does work to set the gateway manually...its hit or miss whether it connects or stays connected
•    I revert to DHCP again and its hit or miss again. Sometimes it finds the gateway on its own and grabs an IP addess and works.
•   Other times it finds the gateway but loses internet connectivity
•   I check and uncheck the bogon and rfc1918 option for the WAN interface then reboot....this usually has not impact.

 
I can not get over my wild experience using this software when all the tutorials make it seem like it should be up and going like it should be ready right after the wizard. That has happened maybe 12 out of the 45 times I have attempted this and was somehow short lived, as it just breaks on its own. I know this software has been used in production but in no way am I feeling confident in using this in my setup at the moment. I am confident in figuring it out though.
 
I refuse to give up on this software. I know there are bugs but this seems like I am missing something. I have reviewed these posts for solutions:
https://www.reddit.com/r/opnsense/comments/szns25/new_to_opnsense_cant_get_wan_ip_via_dhcp/
- Identical issues this time and prior attempts. Both initial setup and once I had successfully acquire an IP address, it would later fall off on its own. I intend on bridging the modem with static IPs utilized but preconfig has been a nightmare so far.
https://forum.opnsense.org/index.php?topic=3529.0
-very similar
-I have not modified my Modem DHCP lease time so that may be a factor
https://forum.opnsense.org/index.php?topic=3455.0
https://github.com/opnsense/core/issues/6301
https://forum.opnsense.org/index.php?topic=22019.0
"What is the problem? The answer to your question is: yes, Realtek is not as good as Intel."
Is it really that bad of an idea to include realtek NIC hardware in my setup?


There are more posts and links that I failed to log but I will continue to update this post as much as I can with a full time job, family, and a troublesome OPNsense router.


Thanks for the help ahead of time. I will try to be in the IRC during the day (CDT time zone)


firewall log still coming
 

[Saarbremer]

Hi,

I see a lot of randomness here and yes, realtek may or may not work. E.g.: Experienced breaking pppoe on realtek before whenever I started bulk operations like downloading more than 10MB sized files.

However, my actual setup runs fine with two realteks.

Anyway what happens if you manually configure the networking to 1000BaseT (on the 2,5) or even 100Base-T. Seems as if something is having trouble working continuously without errors.

I would also simplify the setup to a very simple one: LAN Client connected to OPNsense. Do dhcp and webgui work fine now?

[lew-adv-sol]

I tried to be as detailed as possible but it definitely needs some revising for clarity.

Current setup for testing is onboard Intel nic(LAN) and the 1 realtek usb to rj45(WAN).

I do not see a method to set the speed but I will look into it. Does auto select cause issue within OPNsense?

Funny things is, before I went to bed last night it was still not getting connection on the wan. I shut it down and returned to it this morning to power it on. To my surprise it was getting WAN connections. Nothing had changed.

The only modifications to defaults setting were as described above.
-"I would also simplify the setup to a very simple one: LAN Client connected to OPNsense. Do dhcp and webgui work fine now?"-

The disconnection of the WAN and only using the LAN for troubleshooting gives mixed outcomes bepending on the original issue. If LAN wasnt getting IPs dealt out then no, no changed by removing WAN. If I was already getting IP delegation and removed WAN it did not change my ability to access the GUI, good or bad.


Thanks for looking into this.

[lew-adv-sol]

I drove to work (45 mins) and the connection already dropped by the time I got here. WAN connection is no longer on the network and since I am accessing the home network remotely I am SOL for the moment. I set the laptop on dual wifi and LAN so that Id have a fallback for remote access but something went wrong clearly. I know my ISP is up because I am typing this remotely on the office rig @ home.


I'm just thankful I am over being frustrated with this and accepted that its clearly not time to start using OPNsense in production. The simple tutorials do not give any details on the common pitfalls unfortunately it paints a skewed picture.

I rewatched countless tutorials and matched each step....same issues. Same hardware has been used elsewhere.
What logs can I start going through?

[Saarbremer]

Hi,

I am sorry for the poor performance of your setup. It is hard to say what goes wrong here.

In order to continue research I'd check configuration of LAN IF to match DHCP server config on LAN an check that an IP is reliably assigned to a connecting device. If that doesn't work, check cabling, jacks, NICs or configure speed manually. This can be done on Interfaces -> [LAN]

Interface diagnostics and DHCP log could give more insight in case.

Done achieving that repeat the same for WAN. But not as dhcp server. Configure client instead, switch off dhcp server for WAN!

USB is not known to be the most reliable. Sometimes, power configuration can be tricky. Iirc someone mentioned to switch off powerD to get USB to work stable.

Just done thoughts.

[CJ]

I would replace the usb nic with an intel  Another poster was having issues with their setup until they replaced the usb nic with an actual card.  I'd wager that's the source of the majority of your issues.

[lew-adv-sol]

Thank you for contributing your thoughts. Its hard finding feedback on the subject.
@Tron80 -
-I have verified the gateway on the ISP is reliably handing out IPs
 to other device connected directly to the rj45 port and through a eero device set in AP mode. This has been tested with static assignment of the device IP/gateway on the OPNsense and ISP modem for static IP. The non OPNsense box will acquire connection and respond to both configuration as expected. Not so with the OPNsense box. The remainder of your request is a little vague I apologize. Ive tried each setting one by one and in combination. The post goes into more detail if you haven't already taken a look at it. 
The cabling has been tested and swapped out just for sanity purposes
-I have tested the realtek usb rj45 adapter on other systems(running windows) and have reliable uptime relative to the experience with it connected to the OPNsense box.
-USB is a just another fail point and I do hate the idea of doing it this way. I was really hoping to have a testing box to sort out the learning curve of OPNsense. I want to get a Qotom box or some more appropriate for this application but its not in the budget as of yet.
-I am sorry for my difficulty in understanding you message but please correct me on anything I misread. I will say though from what I gathered, your recommendation had already been tested and documented. I still greatly appreciate the time taken to type it out.

In all theory it should be working like a champ. After it didnt I have removed the extra components and only used defaults.

*As of this post, late testing found some possible issues with one of the adapters. I am scrapping the adapter approach. 4 port NIC is being purchased once I decide on one
@CJ - The realtek nics are not going to be used after this headache. I am having to fallback on my mikrotik routers...(as of posting all 3 routing solutions fail at some point.) I am looking at 4 port intel NICs for my medium form factor optiplex 3070 I have laying around. I hope to get a solid build to work with OPNsense. I'll just go the the virtualized route otherwise.
On a tight budget.


So I plugged in both mikrotik and BOTH had issues at some point. All traffic stopped at the mikrotik gateway or DHCP on the mikrotik failed to hand out an IP. I updated the firmware, changed cabling/adapters, and still the same issue.

I give.

*update* the mikrotik 2011's apparently dont let the sfp be used in slave mode. it is what it is. I do have working routers now. I am ordering the 4 port nic and re-approaching OPNsense at a later time. Thanks.

[CJ]

-I have tested the realtek usb rj45 adapter on other systems(running windows) and have reliable uptime relative to the experience with it connected to the OPNsense box.

I'm not surprised it works better on windows.  That's generally what manufacturer's test on.  BSD in general and routers in particular are a lot more picky.

-USB is a just another fail point and I do hate the idea of doing it this way. I was really hoping to have a testing box to sort out the learning curve of OPNsense. I want to get a Qotom box or some more appropriate for this application but its not in the budget as of yet.

You should be able to pick up an Intel PCIe NIC for super cheap.  That would let you have two ports and be enough for testing.  I always recommend people who are getting started with OPNSense to just run a basic WAN/LAN configuration to start with.This allows you to get familiar with everything and then you can slowly add other bits as you have time and ability.

@CJ - The realtek nics are not going to be used after this headache. I am having to fallback on my mikrotik routers...(as of posting all 3 routing solutions fail at some point.) I am looking at 4 port intel NICs for my medium form factor optiplex 3070 I have laying around. I hope to get a solid build to work with OPNsense. I'll just go the the virtualized route otherwise.
On a tight budget.

That should work great for OPNSense.  I'm using something similar for mine exactly for the cost reasons you described and it's been running fine for years.  I do however make sure that I'm running Intel, Mellanox, Chelsio, etc NICs.

So I plugged in both mikrotik and BOTH had issues at some point. All traffic stopped at the mikrotik gateway or DHCP on the mikrotik failed to hand out an IP. I updated the firmware, changed cabling/adapters, and still the same issue.

I give.

*update* the mikrotik 2011's apparently dont let the sfp be used in slave mode. it is what it is. I do have working routers now. I am ordering the 4 port nic and re-approaching OPNsense at a later time. Thanks.

Not sure what you mean as I'm not familiar with mikrotik, but I know a lot of people use them without issue.

[lew-adv-sol]

The virtual method is definitely where I want to go so this is only more motivation. I did find a few 4 port intel cards so I will be resuming that project before too long.

Mikrotik rb2011 arent beefy on throughput but they are reliable. They know how to cut hardware cost for sure. I'm still familiarizing myself with their routerOS. We use them at my job with our clients so I know them well enough to get them deployed.

Hopefully this thread dies off as just a hardware fluke. Thanks for contributing everyone.