Topics

1

General Discussion / Help with USB NIC + LTE Modem for redundant WAN

« on: March 11, 2024, 04:21:43 am »

I'm having an issue with the second WAN. Seems like the device and NIC work fine. It seems to be able to pick up a DHCP config when the router boots. But if I reconnect the cable while opnsense is running I get continuous dhclient attempts, but no actual response. It's almost as if something about the firewall is preventing DHCP on the "wan2" interface.

Code: [Select]

<13>1 2024-03-10T20:56:08-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="27"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for opt3(ue0)
<13>1 2024-03-10T20:56:09-06:00 OPNsense.home.arpa dhclient 69737 - [meta sequenceId="28"] dhclient-script: Reason PREINIT on ue0 executing
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 76044 - [meta sequenceId="1"] dhclient-script: Reason TIMEOUT on ue0 executing
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 77695 - [meta sequenceId="2"] dhclient-script: New IP Address (ue0): 48.X.X.X
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 78719 - [meta sequenceId="3"] dhclient-script: New Subnet Mask (ue0): 255.255.255.0
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 80979 - [meta sequenceId="4"] dhclient-script: New Broadcast Address (ue0): 48.255.255.255
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 82814 - [meta sequenceId="5"] dhclient-script: New Routers (ue0): 48.X.X.X
<13>1 2024-03-10T20:57:28-06:00 OPNsense.home.arpa dhclient 86364 - [meta sequenceId="6"] dhclient-script: New Routers (ue0): 48.X.X.X
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa dhclient 15693 - [meta sequenceId="7"] dhclient-script: Reason FAIL on ue0 executing
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="8"] /usr/local/etc/rc.linkup: ROUTING: entering configure using 'opt3'
<11>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="9"] /usr/local/etc/rc.linkup: ROUTING: not a valid opt3 interface gateway address: 'missing'
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="10"] /usr/local/etc/rc.linkup: ROUTING: configuring inet default gateway on wan
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="11"] /usr/local/etc/rc.linkup: ROUTING: keeping inet default route to 76.X.X.X
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="12"] /usr/local/etc/rc.linkup: ROUTING: configuring inet6 default gateway on wan
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="13"] /usr/local/etc/rc.linkup: ROUTING: keeping inet6 default route to fe80::XXXXXX%igb0
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="14"] /usr/local/etc/rc.linkup: plugins_configure monitor (,WAN2_DHCP)
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="15"] /usr/local/etc/rc.linkup: plugins_configure monitor (execute task : dpinger_configure_do(,WAN2_DHCP))
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="16"] /usr/local/etc/rc.linkup: plugins_configure ipsec (,opt3)
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="17"] /usr/local/etc/rc.linkup: plugins_configure ipsec (execute task : ipsec_configure_do(,opt3))
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="18"] /usr/local/etc/rc.linkup: plugins_configure dhcp ()
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="19"] /usr/local/etc/rc.linkup: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2024-03-10T20:57:31-06:00 OPNsense.home.arpa opnsense 13518 - [meta sequenceId="20"] /usr/local/etc/rc.newwanip: Failed to detect IP for interface opt3
<13>1 2024-03-10T20:57:40-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="21"] /usr/local/etc/rc.linkup: plugins_configure dns ()
<13>1 2024-03-10T20:57:40-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="22"] /usr/local/etc/rc.linkup: plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2024-03-10T20:57:40-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="23"] /usr/local/etc/rc.linkup: plugins_configure dns (execute task : unbound_configure_do())
<13>1 2024-03-10T20:57:49-06:00 OPNsense.home.arpa opnsense 34870 - [meta sequenceId="24"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for opt3(ue0)
<27>1 2024-03-10T20:57:49-06:00 OPNsense.home.arpa dhclient 68039 - [meta sequenceId="25"] connection closed
<26>1 2024-03-10T20:57:49-06:00 OPNsense.home.arpa dhclient 68039 - [meta sequenceId="26"] exiting.
<13>1 2024-03-10T20:57:50-06:00 OPNsense.home.arpa opnsense 39292 - [meta sequenceId="27"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for opt3(ue0)

It just repeats these attempts in an endless loop until I disable the interface or reboot.

EDIT: Manually running dhclient on the interface gives the same results.

Code: [Select]

root@OPNsense:~ # dhclient -d ue0
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 5
DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 14
DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 16
My address (48.X.X.X) was re-added
My address (48.X.X.X) was deleted, dhclient exiting

Thank you for your time.

2

22.1 Legacy Series / PSA: APU 2 -- BIOS/Firmware update required -- mountroot, error 19

« on: March 01, 2022, 04:03:08 am »

I wanted to put this online somewhere, because my search results were almost entirely unhelpful. I hope this save someone some pain.

If you have a PC Engines APU 2, and you've had it for a while, you MUST update the firmware/BIOS on it to install OPNsense 22.1.

=== Some History (because I want to tell the story) ===
Long ago, in 2015, I tried OPNsense when it first forked, it wasn't for me. I had issues with my multiple VPN setup. I don't know if it was me or the software, but we weren't getting along. I decided to save my self some hours tinkering every week and go back to pfsense.

In December, I was unable to upgrade my existing pfSense from 2.4.5, it seems I encountered a pretty bad (yet common) bug, and the entire package system was corrupted with the wrong versions of things. I tried upgrading it "manually" via USB stick, and it left me with a non-functioning system. Thankfully a downgrade worked, and I had a router again. The corruption left me with a system I couldn't update, and would need to again manually re-install.

I finally had time again, and I've been wanting to try OPNsense again (thanks reddit!), what better time than now? I'm going to have to do a fresh install anyway.

=== Install Problems ===
My hardware is a PC Engines APU 2C4 I got quite some time ago. I think the firmware on it was dated 2016. I may have upgraded it once since I got it.

I downloaded the latest OPNsense. Used belenaEtcher to create a serial USB installer. Took screenshots and a manual backup of my current pfsense config.xml. And planned my migration over a day. I slept on it and felt good to go forward.

I popped the USB into my APU 2C4. Fired up TeraTerm using a spare windows craptop and my trusty USB RS232-Serial cable, and went to town. Powered off the unit, and booted the USB.

This is where the fun starts! After a few minutes of trying to boot and getting a bunch of "Waiting for CAM" messages, it gives up and says "Mounting from ufs:/dev/sdb failed with error 19". And dumps me out to a "mountroot>" command prompt.

It turns out that older versions of the APU firmware don't like using GPT, and the USB 3.0 drivers exacerbate the issue.

=== The Wrong Solution ===
At this point there was a lot of "advice" online to boot setting kern.cam.boot_delay="10000"
This is bad advice! It will make you wait forever for your device to figure out why it can't negotiate a USB 3.0 connection, and it doesn't solve the underlying cause/issue.

=== Problem Solving (or how I stubbornly used Windows) ===
After a bunch of research and looking into BSD and other help forums, I learned about the GPT issue. I finally figured out I really should upgrade the APU firmware, and that would (hopefully) solve the problem.

This is a very good resource on how to use flashrom via software you probably already have installed on your APU!
https://teklager.se/en/knowledge-base/apu-bios-upgrade/

The easiest option would have been to boot back into pfsense, and run flashrom via an SSH session. This didn't work for me because of the corrupt package versioning. I couldn't get the correct versions of packages, and this left me unable to run flashrom. I kept getting 'Shared object "libarchive.so.7" not found, required by "pkg"'. And after using the pkg-static to install flashrom, running it I would get a 'Undefined symbol "fstat@FBSD_1.5"'. These are due to the corrupt packages and versions in pfsense.

Another trip on the merry-go-round! I tried to create a USB 2.0 stick with MBR, TinyCore, and the APU firmware to flash. I couldn't figure out how to make a proper TinyCore USB using belenaEtcher. I'll spare you the details, I just used the PC Engines provided imager.

=== (my) SOLUTION ===
Ultimately, I created a USB 2.0 stick with MBR using Windows. I had to "undo" the GPT partition from a prior OPNsense attempt.
There was a bit of trial an error, the PC Engines imager doesn't partition the drive. I had to make sure it was MBR and FAT32 before starting.
https://docs.microsoft.com/en-us/windows-server/storage/disk-management/change-a-gpt-disk-into-an-mbr-disk
Then ran the PC Engines installer for TinyCore
https://pcengines.ch/howto.htm#TinyCoreLinux
Then loaded the latest APU firmware image onto it. (apu2 v4.15.0.3)
https://pcengines.github.io/

I booted, and ran the flashrom command with no issues.
flashrom -w /media/TINYCORE/apu2_v4.15.0.3.rom -p internal

After that, the etcher created USB with GPT for OPNsense 22.1 worked! No fancy kernel boot parameters required. It didn't take forever to boot either.

=== Conclusion ===
THOU SHALT UPDATE THINE FIRMWARE!

PC Engines is really awesome for continuing to support their systems after all these years. The fact that I could get a firmware that was released just 12 days ago, is mindblowing! I'm really happy with this hardware and the continued support. I'm also happy that it solved the problems. I will definitely be paying attention to firmware updates from now on.

3

19.1 Legacy Series / [Solved] VPN Client restarting -- event_wait : Interrupted system call (code=4)

« on: June 12, 2019, 05:22:28 am »

Sorry if this has been asked before, but I'm at my whits end with this.

I had a setup with OPNsense as my core router/gateway, and it worked. I recently moved and no longer have that option. I'm stuck behind an ISP provided router.

I've made several modifications to the VPN configurations, I've tried a fresh (re)install of opnsense, I've tried a different VPN provider on a different port and different client.

I can get the VPN to work from my phone and from my laptop. So the issue isn't with the VPN provider getting through my ISP.

I've made my opnsense the "DMZ" host for my ISPs router. I've enabled port forwarding as well, and I've disabled the ISP firewall entirely.

Even with all this I can't get the VPN client to stay up. It will connect, authenticate, handshake.... Wait... Then it somehow gets an "interrupt" and exits. At this point the service restarts under a new child pid, rinse and repeat.

Do I need to set a client IP that reflects my WAN IP? I'm running pi.hole on my network so DNS and DHCP service on opnsense is disabled. How do I get opnsense to get the WAN IP from behind a NAT? (like ddclient)

Attached is an example log. Note that it successfully connects, pulls routes, then decides to quit, teardown, and start all over.