Topics

I could use a little help with figuring out terminal modes.

I ssh from powershell to opnsense, and invoke the shell. If I move my cursor to the start of the line (CTRL^A) and start typing, it doesn't redraw the line. If I set the terminal to vt100 (setenv TERM vt100) this seems to fix the issue. However, it makes vi unusable. As in insert mode, it doesn't draw new lines or refresh properly.

Is there something I can do to make the default xterm-256color work? or perhaps make vi work with vt100 mode?

I just don't understand enough about terminal modes.

This is running on a pcengines 2c4 and I do use the serial cable if I need to get local access to the system. The serial terminal all seems to work just fine. It's just when using SSH.

I'm having some issues with multi-wan failover using IPv6.

WAN is DHCPv6

WAN2 is set to SLAAC via LTE Modem, I'm not as concerned that this doesn't seem to work for ipv6 at the moment. I've been able to get things to work with NPT, but I think I will assign dedicated NAT addresses in the future, because NPT needs to also be updated every time the WAN prefix changes.

LAN is set to "Track Interface"

I'm using radv "Router Advertisements" in the services.
It set to "Assisted" with "Automatic" source address.
I've not set any advanced options, everything is default.

I tested my failover by unplugging the cable from my cable modem. When service was restored, the gateway monitoring functioned, and fail to LTE was fine. Once I plugged it back in, I noticed a lot of delay trying to get to test-ipv6.com

I had the same issue on my phone. I cycled the wifi connection on and off and it solved it.

I can see from a windows client that I still have the old prefix/address.

What can I do to solve this issue? I'd like to have clients properly deprecate/abandon an old address when the WAN flaps.

Here is an example output from ifconfig. The c881 is the new address, and the c800 is the old one.

Code Select Expand

Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . : home.arpa
   IPv6 Address. . . . . . . . . . . : 2601:281:c881:fb80:1a7a:5927:4cd4:e21b
   IPv6 Address. . . . . . . . . . . : 2601:281:c800:3910:3e2f:a436:d203:d072
   Temporary IPv6 Address. . . . . . : 2601:281:c800:3910:a512:d226:8873:46cb
   Temporary IPv6 Address. . . . . . : 2601:281:c881:fb80:a857:a7c4:21fe:3929
   Link-local IPv6 Address . . . . . : fe80::d1fd:217e:6ec2:961%25
   IPv4 Address. . . . . . . . . . . : 192.168.1.161
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.1
Should I set radv lifetimes to something more aggressive than the defaults?

I'm trying to write a set of monit instructions to restart unbound if I start to see "SERVFAIL".
This condition seems to happen in my environment when there are WAN issues or there is a Multi-WAN failover.

It appears that "custom" creates a program service checker, but I'm unable to use the "content" keyword to check the output. The "content" rule type is giving an error that the service is not of type "file".

Monit docs say "content" is supported by the "program" service.
https://mmonit.com/monit/documentation/monit.html#PROGRAM-OUTPUT-CONTENT-TEST

In monit, I have a "service" called "dns_resolve_failure" that has the following config:

Code Select Expand

Type   Custom
Path   "/usr/bin/drill google.com"
Start  "/usr/local/sbin/configctl unbound start"
Stop   "/usr/local/sbin/configctl unbound stop"
Tests  "Nothing Selected"
I have a "service test" called "DNS_SERVFAIL" that looks like this:

Code Select Expand

Condition:  content = "SERVFAIL"
Action:     Restart
The services works fine, I can see the output of drill, but I'm unable to assign a "content" test to it.

I'm running OPNsense 25.7.11_9 currently.

I guess I could set up an "advanced" script in "/usr/local/etc/monit.opnsense.d/", but I was hoping I could do it through the GUI.

I'm trying to figure out how to get IPv6 failover working on OPNsense.
My network is configured for dual stack 4 and 6.

WAN1 is Comcast, set to DHCP6 with working prefix deligation, LAN is set to track interface, and IPv6 is working fine.
Clients using SLAAC, and RA is configured "Unmanaged" for the same. I run pi-hole for DNS and local name resolution. All works fine.

WAN2 is t-mobile on an LTE modem, IPv6 on the modem works. The WAN2 interface is set to DHCP6 and getting assigned a /64. Interestingly this connection is also creating a opt5_stf interface, which I've left unassigned, because I don't know what it is.

None of the clients seem to pick up the WAN2 address space. I assume that is because track interface will only use RA for the default gateway. Do I need to set up more RA for another route/subnet?

I have a gateway group for both IPv4 gateways, and IPv6 gateways. The IPv4 failover works fine (no surprise). IPv6 is unable to route once WAN1 goes down.
I can't figure out NPT, the documentation is hazy to me after having configured a few things I thought would work.
pfsense docs mention the target should be /64 (but not the delegated address). How do I know which address that is?
https://docs.netgate.com/pfsense/en/latest/recipes/multiwan-ipv6.html
opnsense docs mention nothing of the sort.
https://docs.opnsense.org/manual/nptv6.html#nptv6

What am I supposed to put in NPT config to get this to work? Where do I find the correct information if the interface address isn't correct? Do I get it from upstream on the modem?

Thanks

I'm having an issue with the second WAN. Seems like the device and NIC work fine. It seems to be able to pick up a DHCP config when the router boots. But if I reconnect the cable while opnsense is running I get continuous dhclient attempts, but no actual response. It's almost as if something about the firewall is preventing DHCP on the "wan2" interface.

Code Select Expand


<13>1 2024-03-10T20:56:08-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="27"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for opt3(ue0)
<13>1 2024-03-10T20:56:09-06:00 OPNsense.home.arpa dhclient 69737 - [meta sequenceId="28"] dhclient-script: Reason PREINIT on ue0 executing
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 76044 - [meta sequenceId="1"] dhclient-script: Reason TIMEOUT on ue0 executing
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 77695 - [meta sequenceId="2"] dhclient-script: New IP Address (ue0): 48.X.X.X
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 78719 - [meta sequenceId="3"] dhclient-script: New Subnet Mask (ue0): 255.255.255.0
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 80979 - [meta sequenceId="4"] dhclient-script: New Broadcast Address (ue0): 48.255.255.255
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 82814 - [meta sequenceId="5"] dhclient-script: New Routers (ue0): 48.X.X.X
<13>1 2024-03-10T20:57:28-06:00 OPNsense.home.arpa dhclient 86364 - [meta sequenceId="6"] dhclient-script: New Routers (ue0): 48.X.X.X
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa dhclient 15693 - [meta sequenceId="7"] dhclient-script: Reason FAIL on ue0 executing
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="8"] /usr/local/etc/rc.linkup: ROUTING: entering configure using 'opt3'
<11>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="9"] /usr/local/etc/rc.linkup: ROUTING: not a valid opt3 interface gateway address: 'missing'
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="10"] /usr/local/etc/rc.linkup: ROUTING: configuring inet default gateway on wan
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="11"] /usr/local/etc/rc.linkup: ROUTING: keeping inet default route to 76.X.X.X
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="12"] /usr/local/etc/rc.linkup: ROUTING: configuring inet6 default gateway on wan
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="13"] /usr/local/etc/rc.linkup: ROUTING: keeping inet6 default route to fe80::XXXXXX%igb0
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="14"] /usr/local/etc/rc.linkup: plugins_configure monitor (,WAN2_DHCP)
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="15"] /usr/local/etc/rc.linkup: plugins_configure monitor (execute task : dpinger_configure_do(,WAN2_DHCP))
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="16"] /usr/local/etc/rc.linkup: plugins_configure ipsec (,opt3)
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="17"] /usr/local/etc/rc.linkup: plugins_configure ipsec (execute task : ipsec_configure_do(,opt3))
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="18"] /usr/local/etc/rc.linkup: plugins_configure dhcp ()
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="19"] /usr/local/etc/rc.linkup: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2024-03-10T20:57:31-06:00 OPNsense.home.arpa opnsense 13518 - [meta sequenceId="20"] /usr/local/etc/rc.newwanip: Failed to detect IP for interface opt3
<13>1 2024-03-10T20:57:40-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="21"] /usr/local/etc/rc.linkup: plugins_configure dns ()
<13>1 2024-03-10T20:57:40-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="22"] /usr/local/etc/rc.linkup: plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2024-03-10T20:57:40-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="23"] /usr/local/etc/rc.linkup: plugins_configure dns (execute task : unbound_configure_do())
<13>1 2024-03-10T20:57:49-06:00 OPNsense.home.arpa opnsense 34870 - [meta sequenceId="24"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for opt3(ue0)
<27>1 2024-03-10T20:57:49-06:00 OPNsense.home.arpa dhclient 68039 - [meta sequenceId="25"] connection closed
<26>1 2024-03-10T20:57:49-06:00 OPNsense.home.arpa dhclient 68039 - [meta sequenceId="26"] exiting.
<13>1 2024-03-10T20:57:50-06:00 OPNsense.home.arpa opnsense 39292 - [meta sequenceId="27"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for opt3(ue0)


It just repeats these attempts in an endless loop until I disable the interface or reboot.

EDIT: Manually running dhclient on the interface gives the same results.

Code Select Expand


root@OPNsense:~ # dhclient -d ue0
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 5
DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 14
DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 16
My address (48.X.X.X) was re-added
My address (48.X.X.X) was deleted, dhclient exiting


Thank you for your time.

I wanted to put this online somewhere, because my search results were almost entirely unhelpful. I hope this save someone some pain.

If you have a PC Engines APU 2, and you've had it for a while, you MUST update the firmware/BIOS on it to install OPNsense 22.1.

=== Some History (because I want to tell the story) ===
Long ago, in 2015, I tried OPNsense when it first forked, it wasn't for me. I had issues with my multiple VPN setup. I don't know if it was me or the software, but we weren't getting along. I decided to save my self some hours tinkering every week and go back to pfsense.

In December, I was unable to upgrade my existing pfSense from 2.4.5, it seems I encountered a pretty bad (yet common) bug, and the entire package system was corrupted with the wrong versions of things. I tried upgrading it "manually" via USB stick, and it left me with a non-functioning system. Thankfully a downgrade worked, and I had a router again. The corruption left me with a system I couldn't update, and would need to again manually re-install.

I finally had time again, and I've been wanting to try OPNsense again (thanks reddit!), what better time than now? I'm going to have to do a fresh install anyway.

=== Install Problems ===
My hardware is a PC Engines APU 2C4 I got quite some time ago. I think the firmware on it was dated 2016. I may have upgraded it once since I got it.

I downloaded the latest OPNsense. Used belenaEtcher to create a serial USB installer. Took screenshots and a manual backup of my current pfsense config.xml. And planned my migration over a day. I slept on it and felt good to go forward.

I popped the USB into my APU 2C4. Fired up TeraTerm using a spare windows craptop and my trusty USB RS232-Serial cable, and went to town. Powered off the unit, and booted the USB.

This is where the fun starts! After a few minutes of trying to boot and getting a bunch of "Waiting for CAM" messages, it gives up and says "Mounting from ufs:/dev/sdb failed with error 19". And dumps me out to a "mountroot>" command prompt.

It turns out that older versions of the APU firmware don't like using GPT, and the USB 3.0 drivers exacerbate the issue.

=== The Wrong Solution ===
At this point there was a lot of "advice" online to boot setting kern.cam.boot_delay="10000"
This is bad advice! It will make you wait forever for your device to figure out why it can't negotiate a USB 3.0 connection, and it doesn't solve the underlying cause/issue.

=== Problem Solving (or how I stubbornly used Windows) ===
After a bunch of research and looking into BSD and other help forums, I learned about the GPT issue. I finally figured out I really should upgrade the APU firmware, and that would (hopefully) solve the problem.

This is a very good resource on how to use flashrom via software you probably already have installed on your APU!
https://teklager.se/en/knowledge-base/apu-bios-upgrade/

The easiest option would have been to boot back into pfsense, and run flashrom via an SSH session. This didn't work for me because of the corrupt package versioning. I couldn't get the correct versions of packages, and this left me unable to run flashrom. I kept getting 'Shared object "libarchive.so.7" not found, required by "pkg"'. And after using the pkg-static to install flashrom, running it I would get a 'Undefined symbol "fstat@FBSD_1.5"'. These are due to the corrupt packages and versions in pfsense.

Another trip on the merry-go-round! I tried to create a USB 2.0 stick with MBR, TinyCore, and the APU firmware to flash. I couldn't figure out how to make a proper TinyCore USB using belenaEtcher. I'll spare you the details, I just used the PC Engines provided imager.

=== (my) SOLUTION ===
Ultimately, I created a USB 2.0 stick with MBR using Windows. I had to "undo" the GPT partition from a prior OPNsense attempt.
There was a bit of trial an error, the PC Engines imager doesn't partition the drive. I had to make sure it was MBR and FAT32 before starting.
https://docs.microsoft.com/en-us/windows-server/storage/disk-management/change-a-gpt-disk-into-an-mbr-disk
Then ran the PC Engines installer for TinyCore
https://pcengines.ch/howto.htm#TinyCoreLinux
Then loaded the latest APU firmware image onto it. (apu2 v4.15.0.3)
https://pcengines.github.io/

I booted, and ran the flashrom command with no issues.
flashrom -w /media/TINYCORE/apu2_v4.15.0.3.rom -p internal

After that, the etcher created USB with GPT for OPNsense 22.1 worked! No fancy kernel boot parameters required. It didn't take forever to boot either.

=== Conclusion ===
THOU SHALT UPDATE THINE FIRMWARE!

PC Engines is really awesome for continuing to support their systems after all these years. The fact that I could get a firmware that was released just 12 days ago, is mindblowing! I'm really happy with this hardware and the continued support. I'm also happy that it solved the problems. I will definitely be paying attention to firmware updates from now on.

Sorry if this has been asked before, but I'm at my whits end with this.

I had a setup with OPNsense as my core router/gateway, and it worked. I recently moved and no longer have that option. I'm stuck behind an ISP provided router.

I've made several modifications to the VPN configurations, I've tried a fresh (re)install of opnsense, I've tried a different VPN provider on a different port and different client.

I can get the VPN to work from my phone and from my laptop. So the issue isn't with the VPN provider getting through my ISP.

I've made my opnsense the "DMZ" host for my ISPs router. I've enabled port forwarding as well, and I've disabled the ISP firewall entirely.

Even with all this I can't get the VPN client to stay up. It will connect, authenticate, handshake.... Wait... Then it somehow gets an "interrupt" and exits. At this point the service restarts under a new child pid, rinse and repeat.

Do I need to set a client IP that reflects my WAN IP? I'm running pi.hole on my network so DNS and DHCP service on opnsense is disabled. How do I get opnsense to get the WAN IP from behind a NAT? (like ddclient)

Attached is an example log. Note that it successfully connects, pulls routes, then decides to quit, teardown, and start all over.