Messages

I ssh from powershell to opnsense

What happens when you use PuTTY instead ?!

I don't have PuTTY installed. I tried a "bash on windows" (MSYS2) terminal, and it has the same behavior as powershell.

I tried terminal from an ubuntu system I have, and "command line editing" does work. VI does not rerender properly.

I can probably live with setting it to vt100 to get cli working well. I rarely have to edit a config. It would just be nice if it worked.

Sorry, I don't understand enough about terminals to solve this issue. AI seems to think this is a common problem and has something to do with how input and interpretations differ between linux/windows and bsd. Remapping backspace is a common suggestion, but the mapping appears to already be there. I think there are a few more differences. For what it's worth, powershell can SSH to any of my linux systems and rewrite cli and render vim properly. This only seems to be an issue with the one BSD system I have (opnsense).

Thanks.

Yes. I use CTRL+A and CTRL+E quite a lot in terminal shells. Often to get to the front or end of line to change the command.

For example going from using ls/ll to view things, then just hitting up-arrow and ctrl+a to change the command to vi, or rm, or something else I want to do with the file I've discovered.

This "line editing" command strategy with the terminal doesn't seem to redraw the line unless I set TERM=vt100.

I also use tab completion. But that seems to work just fine.

How are you getting a vi mode? The shell used on OPNsense is not bash or ksh or sh, at least it doesn't understand 'set -o vi'.

Specifically, I ssh to root@opnsense, next I select "8" for shell. This launches tcsh.
From there I can enter any commands I care to, e.g. `configctl restart unbound`. If I need to edit a config file, I type "vi config.conf".

The rest is simply using vi. For example pressing "i" for "insert" mode and making my changes.

I could use a little help with figuring out terminal modes.

I ssh from powershell to opnsense, and invoke the shell. If I move my cursor to the start of the line (CTRL^A) and start typing, it doesn't redraw the line. If I set the terminal to vt100 (setenv TERM vt100) this seems to fix the issue. However, it makes vi unusable. As in insert mode, it doesn't draw new lines or refresh properly.

Is there something I can do to make the default xterm-256color work? or perhaps make vi work with vt100 mode?

I just don't understand enough about terminal modes.

This is running on a pcengines 2c4 and I do use the serial cable if I need to get local access to the system. The serial terminal all seems to work just fine. It's just when using SSH.

I suppose I could have figured out how to get things to work with pihole. You are right, it is "just dnsmasq" under the hood.

I had a really bad time with it for quite a while. There would be weird problems to troubleshoot, and sometimes unbound would die on the router, and pihole would just get SERVFAIL until I restarted the router. In the end this turned out to be an issue with how I was doing multiwan and RA. Since upgrading opnsense to 26.X and changing a lot of how my multiwan worked (and gaining some more understanding about SLAAC for the LTE backup) things are a lot more stable.

Eliminating the pihole not only helped me simplify things, but also helped me get to the root cause of my network issues.

Today, I'm using unbound for dns (DoT only on quad9), forwarding to opnsense dnsmasq for local lookups. I migrated away from ISC in favor of dnsmasq for IPv6. As KEA isn't quite ready, and it was much simpler to configure DHCP for dnsmasq. That turned out to be the easiest way to get local IPv6 name resolution working again.

Sorry to resurrect this thread. I just thought it may be useful for someone in the future if they stumble upon this.

Just one final update. Sorry to bump this thread. But I wanted to get the info out there.

I've simply decided to ignore having any IPv6 on the backup LTE via SLAAC. This has simplified the gateway monitoring, routes, and switching.

I've noticed that part of the issue with the LTE modem is the ISP will drop the IPv6 connection if any traffic originates from it that isn't from the assigned /64 SLAAC address. This is part of why NPT wasn't great. I could set up NDP in the future to provide addresses for the SLAAC interface, but that just doesn't seem worth it for the few hours per month the backup is in play.

The prefix issue still exists, but is far less catastrophic to the network. The short times appear to help, but I still notice some Windows clients getting stuck with the old prefix and therefore unable to route IPv6 traffic after the primary WAN is restored. I could probably detect and solve this problem with a script, but I'm pretty busy and my family probably doesn't have the patience for that.

Thanks for the great product and support.

Thanks for the reply. There were a few changes I made and things appear to be more stable. I'll have to wait a bit longer to be sure. Over the last few months I've been having to power cycle the router to fix a "split brain" situation with the networks.

The 2 things that seem to have made a difference.

1 - In verifying my settings, I didn't have any DNS server set in [system] > [settings] > [general]. I've corrected this and set them to the same servers as I have in the gateway monitoring.

2 - I've set more aggressive timeouts for RADV. I'm now using:

Code Select Expand

Minimum Interval = 10
Maximum Interval = 30
AdvPreferredLifetime = 60
AdvRouteLifetime = 90
I've disabled any settings for NPTv6 from the [Firewall] > [NAT], because of my GUA prefix changing. Thanks for your feedback about ULAs. I know there are issues with dual stack networks, and it seems this would be one of those cases where using ULA for NPT would simply result in no IPv6 going out the secondary WAN due to "happy eyeballs" and IPv4 preference.

I'll look into NAT66. I'm also researching using a reserved GUA (like 2000:db8:: ) for NPT, but this would be a hack.

Thanks for the time and feedback.

I'm having some issues with multi-wan failover using IPv6.

WAN is DHCPv6

WAN2 is set to SLAAC via LTE Modem, I'm not as concerned that this doesn't seem to work for ipv6 at the moment. I've been able to get things to work with NPT, but I think I will assign dedicated NAT addresses in the future, because NPT needs to also be updated every time the WAN prefix changes.

LAN is set to "Track Interface"

I'm using radv "Router Advertisements" in the services.
It set to "Assisted" with "Automatic" source address.
I've not set any advanced options, everything is default.

I tested my failover by unplugging the cable from my cable modem. When service was restored, the gateway monitoring functioned, and fail to LTE was fine. Once I plugged it back in, I noticed a lot of delay trying to get to test-ipv6.com

I had the same issue on my phone. I cycled the wifi connection on and off and it solved it.

I can see from a windows client that I still have the old prefix/address.

What can I do to solve this issue? I'd like to have clients properly deprecate/abandon an old address when the WAN flaps.

Here is an example output from ifconfig. The c881 is the new address, and the c800 is the old one.

Code Select Expand

Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . : home.arpa
   IPv6 Address. . . . . . . . . . . : 2601:281:c881:fb80:1a7a:5927:4cd4:e21b
   IPv6 Address. . . . . . . . . . . : 2601:281:c800:3910:3e2f:a436:d203:d072
   Temporary IPv6 Address. . . . . . : 2601:281:c800:3910:a512:d226:8873:46cb
   Temporary IPv6 Address. . . . . . : 2601:281:c881:fb80:a857:a7c4:21fe:3929
   Link-local IPv6 Address . . . . . : fe80::d1fd:217e:6ec2:961%25
   IPv4 Address. . . . . . . . . . . : 192.168.1.161
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.1
Should I set radv lifetimes to something more aggressive than the defaults?

I haven't opened a bug ticket, but I took the time to write a custom monit rule file. It wasn't difficult. This suffices for now.

Code Select Expand

root@OPNsense:~ # cat /usr/local/etc/monit.opnsense.d/unbound.conf
check program unbound_servfail with path "/usr/bin/drill google.com" timeout 60 seconds
        start program = "/usr/local/sbin/configctl unbound start" with timeout 30 seconds
        stop program = "/usr/local/sbin/configctl unbound stop"
        if content = "SERVFAIL" then alert
        if content = "SERVFAIL" then restart
        if 5 restarts within 5 cycles then unmonitor
Thanks!

https://imgur.com/a/opnsense-test-content-not-allowed-RrRdwPn
When clicking "save" I get the error on the field. "Test DNS_SERVFAIL with type File Content not allowed for this service type"

I had a lot of frustrations with pihole as DNS and DHCP. It didn't really support IPv6 very well, and trying to write rules in opnsense was a nightmare.

I eventually removed pihole entirely. I use Unbound with the hagezi Multi PRO and Threat Intel blocklists. I forward to Dnsmasq which takes care of DHCP and local DNS for both IPv4 and IPv6.

I'm trying to write a set of monit instructions to restart unbound if I start to see "SERVFAIL".
This condition seems to happen in my environment when there are WAN issues or there is a Multi-WAN failover.

It appears that "custom" creates a program service checker, but I'm unable to use the "content" keyword to check the output. The "content" rule type is giving an error that the service is not of type "file".

Monit docs say "content" is supported by the "program" service.
https://mmonit.com/monit/documentation/monit.html#PROGRAM-OUTPUT-CONTENT-TEST

In monit, I have a "service" called "dns_resolve_failure" that has the following config:

Code Select Expand

Type   Custom
Path   "/usr/bin/drill google.com"
Start  "/usr/local/sbin/configctl unbound start"
Stop   "/usr/local/sbin/configctl unbound stop"
Tests  "Nothing Selected"
I have a "service test" called "DNS_SERVFAIL" that looks like this:

Code Select Expand

Condition:  content = "SERVFAIL"
Action:     Restart
The services works fine, I can see the output of drill, but I'm unable to assign a "content" test to it.

I'm running OPNsense 25.7.11_9 currently.

I guess I could set up an "advanced" script in "/usr/local/etc/monit.opnsense.d/", but I was hoping I could do it through the GUI.

If finally works.

Seems to have been something wrong with the interface config.
I unchecked block privates and block bogons, and it's working now.
I'll probably reenable to bogons block at some point if it doesn't break it, but at least it's up and gateway ping is finally responding.

I'm guessing it uses some private local address for something to do with RA and SLAAC from the LM1200.

The final interface config for "VWAN2" is:
Enable [X]
Block private networks [_]
Block bogon networks [_]
IPv4 Configuration Type [DHCP]
IPv6 Configuration Type [SLAAC]
Reject Leases From [192.168.5.1]
Override MTU [X]

Everything else is unset or left to defaults.

I'm sure that I can get NPT working at this point.

Thanks for the support Maurice.

Yes, I'm in the US. I'm not sure if it's using CGNAT, but it probably is. My IPv6 test results show a different public IPv4 than the modem.

I just tested plugging directly into a windows 10 laptop.

Client was assigned both an IPv6 address and a temporary in the same prefix.

I was able to successfully ping 2620:fe::fe

I can try and get a screenshot and include the ipconfig info. I'm not certain yet why opnsense isn't working.

Another interesting observation is that the DNS servers assigned by the modem are
fd00:976a::9,10

I'm not sure why an ISP would use public ULA address. I'm guessing it's the Netgear providing these.

Let me know what other data I should collect.

Thanks!

I didn't. I will try that.

One thing I also failed to disclose is that the WAN interface is heading into a switch for VLANs. So actually, it's VWAN1 and VWAN2. I'm not sure how this is impacting broadcasts, but as far as the upstream modem devices go, they are untagged. The Opnsense side is a "trunk" port with tags for the 2 WAN VLANs.

I did assign the regular WAN interface, but it is not configured due to the VWANs being what I'm interested in.

I'll try directly connecting my old "networking" laptop that I use for this kind of thing.

Again, thank you.