Messages

I have fixed the issue of "lo0", which seems related to my custom "fib". OPNSense don't support fib and the update cause some incompatibility with my custom fib.

The "Uptime" is "2 days, 03:04:04" without crashing, freezes or disconnect.

I think the issue is somehow related to l0:

            lo0  in      1.090 GB/s          1.092 GB/s          554.198 GB
                 out     1.090 GB/s          1.092 GB/s          554.198 GB

I don't think 1GB/s is correct. I'm not sure why that is happening.

I updated to version 26.7. It works for a couple of minutes and then freezes.

The SSH and the web interface stop working, and the entire network goes down. The video output also freezes, making it impossible to type anything.

I'm using a Ryzen 3700X with a Mellanox ConnectX-4 NIC.
I didn't have this bug on the previous version.

I tried some settings like:
- Disable PowerD
- Change hw.acpi.cpu.cx_lowest=C1
- Change machdep.idle=spin

I also noticed that the "Load" is VERY HIGH. Even before any change, the "Load Average" (from top) was 0.1~2.0, the "CPU Load" on OPNSense UI chart was usually around min. 1 max. 5. Now, the Load Average is "16.66,   13.33,    7.24" and the OPNSense UI reports min. 18 max. 81.

[Fixed on 26.7.2_2]

Fixed on 26.7.2_2

Multi WAN with provider dependant prefixes is not really working that great.

Real IPv6 Multi WAN would require a provider independant prefix. (eg routed to you via gre or another tunnel)

The same limitations also exist for IPv4.

I agree. It would be better if I had my own IP and a dedicated /48 GUA prefix.

I'm exploring that option, but none of my ISPs can announce my IP, even with a business contract. I'm afraid that GRE could reduce performance (speed and latency), especially in higher speeds.

To be fair, I already use WireGuard in a setup similar to GRE. I route a /64 from an external server to OPNsense, which is conceptually similar. Its purpose is to reduce latency to Brazil. I have a couple of VPSs connected together, and the end-to-end latency is around 120 ms, compared to about 220 ms when connecting directly with my ISPs.

Faking the host identity on the router via NAT or NPT is a workaround but also has its own limitations.

Your setup is limited by the underlying infrastructure.

I agree that it's a workaround, but it's better than giving multiple GUAs to each client.

---

Side note: I can remove the ISP router and connect the ONT directly to OPNsense, which would give me the full /56 prefix. However, that ISP uses PPPoE, and I don't think my hardware can handle 10 Gbps over PPPoE. I'm also considering buying another router with PPPoE hardware acceleration and DHCPv6-PD support, then using the following setup: ISP → ONT → My Router (PPPoE + DHCPv6-PD) → OPNsense. That should perform as well as my other two ISPs while still giving OPNsense the delegated IPv6 prefixes. However, I need to buy another router and ONT and pray for the best. Currently, I don't have "ONT → My Router", but a single All-In-One device from ISP (which is a ONT, Router, Switch and Access-Point/WIFI).

 

For an ordinary WAN-to-LAN scenario, NPTv6 should not be necessary. ndp-proxy-go can extend the real on-link GUA prefix to the downstream LAN, allowing clients to use addresses from 2a12:1111:1111:c201::/64 directly.

I understand that NPTv6 is not necessary. I use NPTv6 for multi-wan. In my case I don't want to expose the GUA. If I expose every GUA, from each WAN/ISP, each client will need to balance and fallback on their own. In other word:

A) With GUA, VM_1 will get:
2a12:1111:1111:c201::da4d // from ISP 1
2222:1111:1111:c201::da4d // from ISP 2
2003:1111:1111:c201::da4d // from ISP 3

B) With NPTv6, VM_1 will get:
fdaa:1111:1111::da4d


The case A works with ndp-proxy-go. But that is terrible. Every device/VM will use a single ISP/"WAN", no loading balance, no routing (forcing specific ISP). Also, in some cases fallback doesn't work too, it gets stuck using a single GUA. Every client/VM is responsible for everything.

The case B is easier. If I change the Gateway in OPNSense, everyone will use it. The responsibility relies on OPNSense, which is a single point. I can also force some connections (to specific external IP) to use specific Gateway/ISP, without touching on each device/client.

The issue here is: one specific WAN/ISP don't support DHCPv6-PD. :\

ndp-proxy-go is intentionally stricter. It only answers for host addresses it has actually learned from downstream.

It would be great to have a optional feature that simply answers "that's me" for any address within the configured prefix. I understand that it's not the main goal, but it could be used for backward compatibility with the old ndproxy.

-> Probably the simplest solution would be to not use a proxy, but to just use NAT66, just like using NAT44 (Source NAT). So not a prefix translation, just good old Source NAT Masquerading.

The issue with NAT is the need for port-forwarding, and I can't connect using <my_isp_prefix>::da4d directly. Currently, any connection to ::da4d goes to a specific VM. I can also set up a firewall rule like: 'Allow ANY_WAN to LAN_GAME_SERVERS on port 4000'. This way, anyone can connect to it on a specific port without port-forwarding and each VM has a unique IP.

The proxy is not compatible with NPTv6 setups.

It only proxies single host addresses it discovers on the downstream interfaces, and only answers NDP for these hosts inside the learned prefix.

Have any alternative to it?

I think the old "ndproxy" works with NPTv6. I use ndproxy with wireguard, in another FreeBSD VM, but not on OPNSense.

[WAN_DIGI:]

I'm testing ndp-proxy-go, but it's not working as I expected. I have a multi-WAN setup with NPTv6.

I recently got a new ISP, but their router don't support DHCPv6-PD. I manage to get multiples "/64" in the same interface and I was hopping to route each "/64" to a different LAN.


WAN_DIGI:
2a12:1111:1111:c200::/64
2a12:1111:1111:c201::/64
2a12:1111:1111:c203::/64

LAN:
fdaa:1111:1111::/64 (using NPTv6 from 2a12:1111:1111:c201::/64)

GATEWAY_TEST:
Tier 1: WAN_DIGI
Tier 2: (other WAN)

VM (connected to LAN):
fdaa:1111:1111::da4d (via DHCPv6)


------------

In theory, I was hopping to:

2a12:1111:1111:c201::/64 <-> fdaa:1111:1111::/64

That already works for my other WAN and even Wireguard (which I get /64 externally).

------------

When I test `ping google.com` from VM, the OPNSense shows:

Code Select Expand

16:23:25.354572 00:1b:21:bc:fa:9e > 58:72:c9:40:e8:fc, ethertype IPv6 (0x86dd), length 70: 2a12:1111:1111:c201::da4d > 2a00:1450:4003:809::200e: ICMP6, echo request, id 57938, seq 49995, length 16
However, the router reply:

Code Select Expand

16:23:25.306554 58:72:c9:40:e8:fc > 33:33:ff:00:da:4d, ethertype IPv6 (0x86dd), length 86: fe80::fff1 > ff02::1:ff00:da4d: ICMP6, neighbor solicitation, who has 2a12:1111:1111:c201::da4d, length 32

I don't see any reply to that. Sounds like the ISP router can't find who has that IP.

The NDP Proxy setup is:

Code Select Expand

Enable [X]

Upstream Interface: WAN_DIGI

Downstream Interfaces: LAN

Proxy router advertisements [_] (unchecked)

Install host routes [X] (I already test as unchecked)

The Log File is empty, even with Debug Log enabled.

-----

The documentation states "It can proxy SLAAC on-link prefixes to several downstream interfaces by proxying", so it don't work with DHCPv6? Or I'm missing something?

Maybe the NPTv6 is causing the issue?

I currently have three WAN connections. My primary goal was to use them for failover and basic load balancing.

However, I've noticed that latency and routing change between ISPs on a weekly or monthly basis. Some ISPs have terrible routes to specific servers, especially international ones (for example, Brazil ↔ Europe). I'd like to know whether OPNsense can automatically switch the WAN based on latency.

Does OPNsense provide a feature that can continuously ping a set of IP addresses and choose the WAN with the lowest latency?

For example, consider three WANs: WAN_A WAN_B, and WAN_C.

The idea would be:

* WAN_A → ServerSaoPaulo = 170 ms
* WAN_B → ServerSaoPaulo = 230 ms
* WAN_C → ServerSaoPaulo = 123 ms

In this case, I would want all connections to ServerSaoPaulo to use WAN_C* OPNsense should only switch to WAN_A or WAN_B if WAN_C experiences high latency or significant packet loss.

For connections to ServerSomewhereElse: any of the WANs can be used.

You have to go in the firewall-rule on your WireGuard-interface, which allows the traffic, tick advanced mode and then set the reply-to to your gateway of choice: WireGuard.

I didn't see the 'Advanced' toggle in `Rules [new]` dialog.

Editing each rule and adding Reply-To to 'WAN_WIREGUARD_GW' works. Thank you.

If I send one UDP/TCP/ICMP packet to 2001:my:isp:1111::4242 I get a reply from the same 2001:my:isp:1111::4242.

What does this even mean. You ping your own IP-addresses? This is not a valid test I would guess.

That means that it works, because the reply is from the same IP of the target. If I send a ping to 1.1.1.1 I get a reply from 1.1.1.1.

The issue is: when you ping 1.1.1.1 and get a reply from 2.2.2.2 (because 2.2.2.2 is the other WAN). That happens because of multi-WAN, the output uses another WAN (2.2.2.2) instead of the right WAN (1.1.1.1).


I will clarify, I have the following interfaces:
- WAN_MEO          (i.e 2111::/56)
- WAN_NOS          (i.e 2222::/56)
- WAN_WIREGUARD    (i.e 2333::/56)

I have one LAN:
- LAN               (fdcc::/64)

I'm pinging from the internet (example: https://fra-de-ping.vultr.com):

2001:19f0:6c00:8002:5400:ff:fe00:536a -> 2111::1234 (WAN_MEO) -> fdcc::1234 = Works
2001:19f0:6c00:8002:5400:ff:fe00:536a -> 2222::1234 (WAN_NOS) -> fdcc::1234 = Works
2001:19f0:6c00:8002:5400:ff:fe00:536a -> 2333::1234 (WAN_WIREGUARD) -> fdcc::1234 = Fail

The Wireguard fails because the reply/response uses WAN_MEO or WAN_NOS instead of WAN_WIREGUARD.

Basically, you (in the internet) send ICMP to 2333::1234 and receives a ICMP reply from 2111::1234, of course it doesn't work. I can't find a way to force the exit to match the input. The "reply-to" works for WAN_NOS and WAN_MEO, but doesn't apply to WAN_WIREGUARD.

[Background:]

Background:

I have (almost) IPv6-only network with two WAN from two independent ISP. I use NPTv6 to match the ULA with the external IP.

Everything works fine. If I send one UDP/TCP/ICMP packet to 2001:my:isp:1111::4242 I get a reply from the same 2001:my:isp:1111::4242. If I send to 2001:another:isp:2222::4242, I get a reply from 2001:another:isp:2222::4242. The `reply-to` is working.

In the underground both IPs are translated to fd00::4242 (for instance).

The Wireguard:

I added a Wireguard connection (as client). So, my OPNSense is connected to an external server. That external servers routes the /64 block to Wireguard tunnel and also have a NDP proxy.


Back to OPNSense, I have added the "external server IP" to my NPTv6.

The goal:

Well, I want to use the /64 from the external server. Basically, my server can be reached by 2001:server:ip:2222::4242. That will send the packet to  fd00::4242, similarly to ISP 1 and ISP 2.

In other words, use Wireguard as GRE Tunnel or another ordinary WAN.

The issue:

No matter how hard I try, the packets never routes back to Wireguard. If I send one ping (from another machine) the response will come from ISP 1 or ISP 2 (the other WAN's), but not from the external server's IP.

In other words, the packet comes from WG and leaves from WAN_1.

For some reason, the "WAN" interface (using Wireguard) doesn't have "reply-to" attached. I already have created some rules in the "Rules [new]", forcing to use Wireguard Gateway. I already try checking "Dynamic gateway policy" or creating another Gateway (using the interface), without success.

The packets are always leaking from another Gateway/WAN.

Anyone have any idea of how to force the same behaviour of WAN on Wireguard interfaces?

The new unbound 1.25.1 seems to include some important security fixes. I didn't update it yet, due to this DNS64 bug.

Have any other alternative to Unbound DNS (with DNS64 support)?

Thank you, running `opnsense-revert -r 26.1.7 unbound` worked. :D