Messages

Up Queue

Down Queue

Up Pipe

For completeness, here are mine.

I disagree with a lot of the 'how to' out there, in the following:

- Firstly, I recommend reading the RFC.  It's nicely written and fairly easy to understand, it gives a better understanding of how things can/should be tweaked, than any How To:

https://datatracker.ietf.org/doc/html/rfc8290

- Quantum should not be adjusted, for high bandwidth general use. The default MTU 1500 +14 is 'good'.  The whole '300 per 100Mbits' has been completely misunderstood and is just plain wrong in my view.  Read the RFC. Tweak it below 100Mbits, above don't bother. If you have lots of small packets, maybe, otherwise don't bother.

- Don't change the queues on the pipes, leave them dynamically assigned.
- On the pipe FQ Codel Limit needs to be the maximum simultaneous packets, both directions, managed by the shaper instance.  The value in my screenshots, is the maximum allowed.
- FQ Codel flows, around the maximum states/sessions the firewall is expected to handle, i.e flows
- Delay of 1ms on the pipe, to stop out of order packets at saturation due to broken (?) fastio:

https://redmine.pfsense.org/issues/11192

- I use FQ Codel ECN, on the Pipe and Queue

Attachments to follow.

Would really need to see the Pipe, Queues and Rules - with Advanced Mode toggled - themselves.  Something is certainly not matching on the upload rules, possibly Queue mask is wrong/not set, or Rule direction wrong or not set.

However, FQ - Fair Queue, or Flow Queue - is not really meant for doing 'multitier' on 1 connection - it's for 'fair' sharing, all traffic, amongst all flows, on 1 pipe.

With 2 pipes, 150/150 and 30/15, the firewall expects the total bandwidth available to be 180/165 - i.e there is no overlap and/or subtracting the 30 guest from the 150 total limit.

Whilst it's true that it is possible to restrict the Guest to 30/15, if you're running your main pipe at capacity (150 download, 30 on the guest, simultaneously) it's going to be over saturated.

P.S. There is no source/destination subnet set on the guest rules, also cannot see 'direction' there as well, nor mask on Queue.

nginx can load balance UDP.

https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-udp-load-balancer/

As for implementing in/with opnsense, pass.

Unfortunately not, 8 ports on an integrated board.  Not without replacing the entire thing.

However, this problem does only seem to show when using PPPoE - having another device perform PPPoE (ISP router, Mikrotik) there are no issues with the box/board generally performing DHCP/static IP assignment.

On the plus side, it has now made it to almost 24 hours running PPPoE - one of only a handful of times it has done so - since disabling both MSI/MSI-X.

If the ISP modem is doing something akin to 'half bridge' then it is likely just assigning the IP by DHCP (?) to opnsense - in which case, a shorter lease might help.

However, I would first check that:

- opnsense 'sees' the new IP (Interfaces -> Overview) after it has been assigned by the router, if it doesn't, you possibly need to force a new DHCP request
- If it DOES see the new IP, try reloading the firewall (just reloading the PF service should do it) rather than the entire box, to see if this fixes things.  Although if it does see the new IP, you should be able to see 'newwanip' running and clearing previous states in the logs (unless you have sticky sessions enabled).

Spoke too soon, happened again, albeit after a few days - then happened twice in a few hours.

I installed the Intel em drivers - 7.7.8 - then noticed the below in the log, when it happened again:

Code Select Expand

em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting

Only ever em0 and seemingly queue 0 where PPPoE ends up (due to single queue for PPPoE?)

Bit more Googling, found examples of others using 82574L on Linux, FreeBSD, etc.  A fix for Linux, was to force MSI not MSI-X.

Decided to try disabling all MSI/MSI-X, to go back to legacy IRQ:

Code Select Expand

hw.pci.enable_msix="0"
hw.pci.enable_msi="0"

Now monitoring...and waiting .... again... If this helps, will then try with just MSI-X disabled.

EDIT: Also just spotted this, seems to happen at boot, something else to look into:

Code Select Expand

WARNING: attempt to domain_add(netgraph) after domainfinalize()
ng0: changing name to 'pppoe0'

EDIT 2: Other 'things' to potentially consider, if running solely on legacy IRQ resolves the issue, to then re-enable MSI-X:

- Disabling Hyper-threading, to stop interrupts bouncing between threads
- machdep.disable_msix_migration=1

machdep.disable_msix_migration: Disable migration of MSI-X interrupts between CPUs

For the last few days, the issue seems to have stopped - but still none the wiser, really, as to the cause - final thoughts for now and recent changes:

- 'systat -vmstat 1'

I noticed that the interrupts for 1 em0 queue, which I understand PPPoE will only use 1, even at a fairly low load of 90-100Mbps (both directions concurrently) were reaching 5000-6000, the default seems to be 8000.  I don't know if these scale in a linear fashion based on throughput/packets, but I added:

Code Select Expand

hw.em.max_interrupt_rate: 32000
hw.em.rx_process_limit: -1

- Made sure all CPU scaling/PowerD was disabled and rebooted, difference is negligible anyway as the load is fairly consistent on this firewall, albeit with peak time increases

- No MAC spoofing
- No promiscuous
- MTU 1508 on PPPOE (for 1500 calculated)
- Disabled Flow Control on all interfaces
- Increased Codel flows/limits, these are the defaults so I don't set them on the shaper pipe, no changes to queue on the pipe (let it manage dynamically) and no changes to Quantum (1514, 1500 MTU +14 seems to be the best for high bandwidth, general use)

Code Select Expand


# Some suggestions this should be equal to at least maximum sessions/states, i.e flows, I believe. 
# Firewall regularly has at least 2000 sessions/state entries, so to allow for bursting
net.inet.ip.dummynet.fqcodel.flows: 8192
# The default hard size limit (in unit of packet) of all queues managed by an instance of the scheduler.
# This is the absolute upper limit permitted
net.inet.ip.dummynet.fqcodel.limit: 20480


- Increased the other interface queues:

Code Select Expand

net.inet.ip.intr_queue_maxlen: 2048
net.isr.defaultqlimit: 2048
net.link.ifqmaxlen: 2048 # Set to sum of RX/TX NIC descriptors; default 1024 descriptors
net.route.netisr_maxqlen: 2048

- Whilst TSO was (should have been) already disabled, based on opnsense defaults, also added the below:

Code Select Expand

net.inet.tcp.tso: 0

- Bind threads to CPUs and limit the number of threads (4 in my case):

Code Select Expand

net.isr.bindthreads: 1
net.isr.maxthreads: -1

I have no idea if:

- Something happened at the ISP, ONT upgrades, etc that caused the initial problems. Although it did not impact the ISP router PPPoE if that is/was the case.
- Interrupt/packet processing limit
- Something 'left over' from PowerD and/or CPU scaling, that was unloaded after a reboot/disable

I also did some reading into NetGraph, as:

"The trick part is that after PPPoE session is established, mpd5 does not process its traffic as it goes completely in-kernel"

.. for me, MPD made the connection successfully but it would drop at random points there after - and then refuse to connect until rebooted.

I have also noticed that it takes 6 seconds for the PPPoE connection to get a response from the ISP, then the connection completes in a further 2 seconds:

Code Select Expand

<30>1 2023-11-28T07:50:25+00:00 Firewall.localdomain ppp 75232 - [meta sequenceId="12"] [wan_link0] PPPoE: Connecting to ''
<30>1 2023-11-28T07:50:31+00:00 Firewall.localdomain ppp 75232 - [meta sequenceId="1"] PPPoE: rec'd ACNAME "XXXXX-XXX-C1"
.......
<30>1 2023-11-28T07:50:32+00:00 Firewall.localdomain ppp 75232 - [meta sequenceId="70"] [wan] IFACE: Rename interface ng0 to pppoe0

... and I believe it times out after 9 seconds by default, so a slow responding PPPoE server could contribute, but then I would have expected it to never connect initially (which was not the case).

Not using RSS in sysctl (em NICs seem to use this anyway, based on 'systat -vmstat' output (?) and documentation) or dispatch deferred - whether or not this is still the case, most documentation I found says it should be left on direct if possible especially if there is shaping.

Died again - after about 5-6 hours.

I give up for now, will stick with the ISP router and/or rebuild with something else to test.

Appreciate you taking the time to share your experiences/config.

So far so good - looking hopeful.

I was experimenting with RSS at one point and I note:

Code Select Expand

If RSS is enabled with the 'enabled' sysctl, the packet dispatching policy will move from 'direct' to 'hybrid'.


But I decided to revert back to the defaults of NOT using RSS - and whilst I didn't start seeing PPPoE problems immediately, it's possibly within a week of reverting.

If it is setting to 'non-direct' that resolves this problem, which it's looking likely you might be right, then when I had RSS enabled it would have switched to 'hybrid' potentially having the same/similar impact as 'deferred'.

....but at a loss why this should completely break PPPoE on Direct, until the box is rebooted. 

Documentation on dispatch states 'impacted hardware' and 'performance improvements' - but my hardware has plenty of overhead, that I assumed this wasn't needed..and not wanting to 'tweak settings for the sake of tweaking'!

Much obliged - only difference from my last drop/crash and now, is:

- Forcing a MAC on the physical WAN
- Setting dispatch to deferred

... I can't see how these would make a difference, based on the documentation, but hey.... nothing to lose :)

Dropped again - arse.

> I still suggest that your best bet for understanding what is going on is to get a capture from a monitoring switch between the ONT and your OPNsense.

I hear you, I do, I'm not disagreeing, but at the moment that's not possible without a lot of faff.  However, my symptoms do appear to be different from yours.

But anyway, I added the below:

*ensure that tunables net.isr.dispatch=deferred and net.isr.maxthreads=<number of cores> (and rebooted)
*assign your own MAC to underWAN (and NOT to the interface to the pppoe device)

You just 'force assigned' the physical (underWAN) MAC back to itself on the physical interface?  Or by 'own MAC' you mean the own MAC of the ISP router?

Given the VLAN tagging however, I wouldn't expect the ISP to see the physical Mac - only the VLAN and/or PPPoE?

Well, it is has made it to an hour of uptime - which in recent days testing, it hadn't

I am beginning to wonder if this is a Shared Forwarding problem - which I 'have' to have enabled, to use shaping, as I use Policy Based Routing heavily - as I had previously tried with Promiscuous on the physical port :-/

I have 2 other opnsense setups, which have also seen - I am now beginning to suspect - something similar, where the NIC just hangs or doesn't 'see' packets.  The other 2 setups, do not use PPPoE but do/did have Shared Forwarding enabled - one of the deployments only saw random crash/WAN hang, under semi load, after upgrading from 22.7 -> 23.7 ... and had been rock solid for years prior.

Fingers crossed things will run for a few days, then I can start to narrow down Shared Forwarding vs Promiscuous on the physical.