Messages

Franco,

I recently installed a fresh install of opnsense 26.1 on a router. I have been running dual-wan for years (ATT & Spectrum), but with ISC DHCP for v6/v4, and figured it was time for a fresh config.

The default was DNSMasq, I couldn't get router advertisements to work or see leases (though ipv4 was working, ipv6 was not), so I switched over to KEA / RA. DHCPv4/6 are working well and assigning leases and RA daemon is configured as Managed (A+O) and working great. I get a warning that I should be using a /64 it doesn't seem to effect anything.

I also switched over to the new firewall rules.

My networks are as follows:
WAN1: DHCP / DHCPv6
WAN2: DHCP / DHCPv6
LAN: Static - 172.20.0.1/24 / fd00::1/112
*probably should make it a /120 to match number of addresses, or a /96 so I can 1:1 the values in the last octet to the last group.

** Note: Image tags not working for me, included links **

My Home screen: <link>


NAT is configured manually, and I have the following rules: <link>


I have the following gateways: <link-int> <link-group>



If I wait about couple minutes a link-local IP (fe80 will show up) on WAN1, and I can then manually start the gateway monitor for WAN1_DHCP6: <link>


Firewall: <link1> <link2>



SYSLOG : <link>
Config : <link>
*Password hash removed

I tried the patch and didn't notice any different behavior before.

Post-Patch SYSLOG: <link>

I do have a few opinions on Multi-WAN configs:
- I like using a private range for IPv6 and NAT'ing, because it means IPv4 works exactly the same as IPv6 which makes it simple to manage.
- When traffic is being redirected to different gateways, tracking an interface seems problematic.
- Defaulting to a net in the private IP space (fc00/7), and doing a One-to-One NAT, is probably the best solution when using multiple WAN/Gateways (I personally just NAT to the interface address, but you have the IPs with v6 so might as well use them).

These are just my opinions, but IMHO IPv6 keeps pretending they engineered all the use-cases away for translation, but I just think they cause more problems trying to throw away the toolbox.

In any case, I'd love to get my router's second ISP (Spectrum) up and working, and both of them without manually intervention (hitting start on the gateway monitor).

This is my home router and not a production system, and i haven't added my lab nets yet so it's pretty barebones. If you want me to test anything let me know.

Reading this document (posted elsewhere on the forum) https://datatracker.ietf.org/doc/html/rfc8290 showed me a number of the common recommendations are just, well, wrong.

I ran into a lot of that as well when I was first figuring things out.

FQ-CoDel quantum should be set at your WAN MTU (in my case, 1514 bytes)

100% true, don't know why people would want to split packets up.

FQ-CoDel limit doesn't really need messed with, this setting defines the maximum number of packets that CAN be queued.  The default is 10240.  Most recommendations are to drop this value significantly, thus, causing the console flood messages.  For me, on my download pipe, I left this at the default of 10240.  For my upload pipe (with a max speed of 40Mbps) I halved it to 5120.  No real reason to, but my uploads will never saturate it.

Scaling the limit with bandwidth can help with Bufferbloat. I have ATT 500mbps symetric fiber, and the first hop is the converter box that I have bypassed. With a 2ms target and 4ms interval, 1200 limit is best and I never get any of the warnings. With a higher limit, it has a tendency to burst much higher for a longer interval which increases packet latency.

But yeah, the OP's limit is set too low if he's getting those warnings. It needs to be increased.

Recipe for ATT 500mbps (they over provision to about 640mbps):
Pipe: ATT-IN
Bandwidth: 560
Scheduler Type: fq_codel
Target: 2ms (should be much higher for many, ping your upstream gateway to get good numbers).
Interval: 4ms
Quantum: 1514 (Packet Size, see https://man.freebsd.org/cgi/man.cgi?ipfw)
Limit: 1200 (about perfect for 500mbps, 1500 is worse for me)
*only enable ECN on outgoing pipes/queues.

Pipe: ATT-OUT
Bandwidth: 560
Scheduler Type: fq_codel
Target: 2ms (should be much higher for many, ping your upstream gateway to get good numbers).
Interval: 4ms
Quantum: 1514 (Packet Size, see https://man.freebsd.org/cgi/man.cgi?ipfw)
Limit: 1200 (about perfect for 500mbps, 1500 is worse for me)
ECN: Enabled

Results: https://www.waveform.com/tools/bufferbloat?test-id=1ece2bf6-b7a2-4a40-9817-c90db7eafcd1
Best: 13ms+0+0
Any Day: 14ms+1+1/15ms+0+0

[if appropriate]

I have ATT 500mbps symmetric fiber.

Pipe: ATT-IN
Bandwidth: 540
Scheduler Type: fq_codel
Target: 2ms
Interval: 4ms
Quantum: 1514 (Packet Size, see https://man.freebsd.org/cgi/man.cgi?ipfw)
Limit: 1200 (about perfect for 500mbps)

Pipe: ATT-OUT
Bandwidth: 540
Scheduler Type: fq_codel
Target: 2ms
Interval: 4ms
Quantum: 1514 (Packet Size, see https://man.freebsd.org/cgi/man.cgi?ipfw)
Limit: 1200 (about perfect for 500mbps)
ECN: Enabled

---
Queues:
- HIGH-IN/OUT
- LOW-IN/OUT
Rules:
-IN & OUT rules for pushing ACK packets under 140 bytes to HIGH.
-IN & OUT rules for pushing DNS to High (by IP) to HIGH.
-IN & OUT rules for pushing regular traffic to LOW.

Notes:
-For rules I use the direction parameter to specify in/out instead of source/dest.
-I only do ECN for outgoing packets, because windows and linux now support it, but not every router/system between you and the internet host probably will.
-Quantum should be large enough to fit a packet, limit should be scaled with bandwidth.
-Don't be afraid to play with super low target/intervals with fiber if appropriate. It could be pretty low latency to your first hop. Do a ping to the first gateway and set your target/interval appropriately.

Results: https://www.waveform.com/tools/bufferbloat?test-id=1ece2bf6-b7a2-4a40-9817-c90db7eafcd1
Best: 13ms+0+0,
Any time every day: 14ms+1+1 / 15ms+0+0