PPPoE drop/disconnect, requires a reboot to fix

[iMx]

So far so good - looking hopeful.

I was experimenting with RSS at one point and I note:

Code Select Expand

If RSS is enabled with the 'enabled' sysctl, the packet dispatching policy will move from 'direct' to 'hybrid'.


But I decided to revert back to the defaults of NOT using RSS - and whilst I didn't start seeing PPPoE problems immediately, it's possibly within a week of reverting.

If it is setting to 'non-direct' that resolves this problem, which it's looking likely you might be right, then when I had RSS enabled it would have switched to 'hybrid' potentially having the same/similar impact as 'deferred'.

....but at a loss why this should completely break PPPoE on Direct, until the box is rebooted. 

Documentation on dispatch states 'impacted hardware' and 'performance improvements' - but my hardware has plenty of overhead, that I assumed this wasn't needed..and not wanting to 'tweak settings for the sake of tweaking'!

[iMx]

Died again - after about 5-6 hours.

I give up for now, will stick with the ISP router and/or rebuild with something else to test.

Appreciate you taking the time to share your experiences/config.

[iMx]

For the last few days, the issue seems to have stopped - but still none the wiser, really, as to the cause - final thoughts for now and recent changes:

- 'systat -vmstat 1'

I noticed that the interrupts for 1 em0 queue, which I understand PPPoE will only use 1, even at a fairly low load of 90-100Mbps (both directions concurrently) were reaching 5000-6000, the default seems to be 8000.  I don't know if these scale in a linear fashion based on throughput/packets, but I added:

Code Select Expand

hw.em.max_interrupt_rate: 32000
hw.em.rx_process_limit: -1

- Made sure all CPU scaling/PowerD was disabled and rebooted, difference is negligible anyway as the load is fairly consistent on this firewall, albeit with peak time increases

- No MAC spoofing
- No promiscuous
- MTU 1508 on PPPOE (for 1500 calculated)
- Disabled Flow Control on all interfaces
- Increased Codel flows/limits, these are the defaults so I don't set them on the shaper pipe, no changes to queue on the pipe (let it manage dynamically) and no changes to Quantum (1514, 1500 MTU +14 seems to be the best for high bandwidth, general use)

Code Select Expand


# Some suggestions this should be equal to at least maximum sessions/states, i.e flows, I believe. 
# Firewall regularly has at least 2000 sessions/state entries, so to allow for bursting
net.inet.ip.dummynet.fqcodel.flows: 8192
# The default hard size limit (in unit of packet) of all queues managed by an instance of the scheduler.
# This is the absolute upper limit permitted
net.inet.ip.dummynet.fqcodel.limit: 20480


- Increased the other interface queues:

Code Select Expand

net.inet.ip.intr_queue_maxlen: 2048
net.isr.defaultqlimit: 2048
net.link.ifqmaxlen: 2048 # Set to sum of RX/TX NIC descriptors; default 1024 descriptors
net.route.netisr_maxqlen: 2048

- Whilst TSO was (should have been) already disabled, based on opnsense defaults, also added the below:

Code Select Expand

net.inet.tcp.tso: 0

- Bind threads to CPUs and limit the number of threads (4 in my case):

Code Select Expand

net.isr.bindthreads: 1
net.isr.maxthreads: -1

I have no idea if:

- Something happened at the ISP, ONT upgrades, etc that caused the initial problems. Although it did not impact the ISP router PPPoE if that is/was the case.
- Interrupt/packet processing limit
- Something 'left over' from PowerD and/or CPU scaling, that was unloaded after a reboot/disable

I also did some reading into NetGraph, as:

"The trick part is that after PPPoE session is established, mpd5 does not process its traffic as it goes completely in-kernel"

.. for me, MPD made the connection successfully but it would drop at random points there after - and then refuse to connect until rebooted.

I have also noticed that it takes 6 seconds for the PPPoE connection to get a response from the ISP, then the connection completes in a further 2 seconds:

Code Select Expand

<30>1 2023-11-28T07:50:25+00:00 Firewall.localdomain ppp 75232 - [meta sequenceId="12"] [wan_link0] PPPoE: Connecting to ''
<30>1 2023-11-28T07:50:31+00:00 Firewall.localdomain ppp 75232 - [meta sequenceId="1"] PPPoE: rec'd ACNAME "XXXXX-XXX-C1"
.......
<30>1 2023-11-28T07:50:32+00:00 Firewall.localdomain ppp 75232 - [meta sequenceId="70"] [wan] IFACE: Rename interface ng0 to pppoe0

... and I believe it times out after 9 seconds by default, so a slow responding PPPoE server could contribute, but then I would have expected it to never connect initially (which was not the case).

Not using RSS in sysctl (em NICs seem to use this anyway, based on 'systat -vmstat' output (?) and documentation) or dispatch deferred - whether or not this is still the case, most documentation I found says it should be left on direct if possible especially if there is shaping.

[iMx]

Spoke too soon, happened again, albeit after a few days - then happened twice in a few hours.

I installed the Intel em drivers - 7.7.8 - then noticed the below in the log, when it happened again:

Code Select Expand

em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting
em0: Watchdog timeout Queue[0]-- resetting

Only ever em0 and seemingly queue 0 where PPPoE ends up (due to single queue for PPPoE?)

Bit more Googling, found examples of others using 82574L on Linux, FreeBSD, etc.  A fix for Linux, was to force MSI not MSI-X.

Decided to try disabling all MSI/MSI-X, to go back to legacy IRQ:

Code Select Expand

hw.pci.enable_msix="0"
hw.pci.enable_msi="0"

Now monitoring...and waiting .... again... If this helps, will then try with just MSI-X disabled.

EDIT: Also just spotted this, seems to happen at boot, something else to look into:

Code Select Expand

WARNING: attempt to domain_add(netgraph) after domainfinalize()
ng0: changing name to 'pppoe0'

EDIT 2: Other 'things' to potentially consider, if running solely on legacy IRQ resolves the issue, to then re-enable MSI-X:

- Disabling Hyper-threading, to stop interrupts bouncing between threads
- machdep.disable_msix_migration=1

machdep.disable_msix_migration: Disable migration of MSI-X interrupts between CPUs

[johndchch]

can you switch out the 82574 for something newer?  I had stability issues with the em driver and an 82576 - went away going to something which is more current ( igb, ixgbe etc )

[iMx]

Unfortunately not, 8 ports on an integrated board.  Not without replacing the entire thing.

However, this problem does only seem to show when using PPPoE - having another device perform PPPoE (ISP router, Mikrotik) there are no issues with the box/board generally performing DHCP/static IP assignment.

On the plus side, it has now made it to almost 24 hours running PPPoE - one of only a handful of times it has done so - since disabling both MSI/MSI-X.

[iMx]

Three days, no PPPoE drops, since disabling MSI/MSIX.  Might leave it this way over Christmas/NY, to see if it continues to prove itself.

Box has 8 ports, I'm only using the first 4 - so it doesn't pose much/any of a problem that the IRQ is shared with the last 4 ports

Code Select Expand

vmstat -i
interrupt                          total       rate
irq16: em0 em4+                998150914       3871
irq17: em1 em5                    592811          2
irq18: em2 em6+                311699977       1209
irq19: em3 em7+                  4029536         16
irq22: hdac0                           9          0
irq23: ehci1                      386863          2
cpu0:timer                     277593929       1077
cpu1:timer                     150527204        584
cpu2:timer                     176130400        683
cpu3:timer                     142581455        553
Total                         2061693098       7996

For my benefit as much as anything, next steps I think are:

- Make sure anything 'unused' is disabled in the BIOS, inbuilt audio, etc.
- Did see another report some time ago, on FreeBSD mailing list, where they recommended unloading USB from the kernel to prevent an em(4) oddity/crash/hang and re-testing.  Would prefer to avoid, for recover-ability issues, but can at least make sure USB Legacy is disabled, USB3.0 if it has it, etc.

Then:

- Re-enable both MSIX/MSI
- Disable MSI-X migrations: machdep.disable_msix_migration=1

machdep.disable_msix_migration: Disable migration of MSI-X interrupts between CPUs

If problems still occur, then:

- Try disabling HyperThreading (don't tell Theo)

If problems still occur:

- Move back to just MSI enabled, MSI-X disabled

If problems STILL occur:

- Then we just leave it on legacy IRQ and move on with life!

Or:

- Re-enable the FreeBSD Intel driver, see if it provides the option to disable MSIX on 1 interface em0 (where PPPoE resides) only.

I have only ever been able to reproduce this presumed em(4) lock up, with MSI/MSIX enabled, when using PPPoE - regular interfaces, non-PPPoE, the box has gone for a couple of months without issues.

As PPPoE moves to in-kernel after MPD makes the connection, this maybe suggest some driver/kerne conflict/problem?!?

[iMx]

ISP had an unrelated outage this morning, so they ruined my PPPoE uptime - just over 5 days, this box running PPPoE had never made it that far previously.

Used this as an opportunity to:

- Disable everything ASPM (Active State Power Management) in the BIOS. pciconf confirms disabled:

Code Select Expand

pciconf -lcv | grep -i asp
                 link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)

- Disable all the various C states lower than C1 (C2, C6, C7, etc), shouldn't have been used anyway
- Disabled Azalia (on board audio)
- Set 'Legacy USB' to 'Auto' (legacy USB will then only be enabled, at boot, if there is a keyboard, mouse, etc plugged in)
- Disable USB Mass Storage
- Re-enabled MSI, left MSI-X disabled:

Code Select Expand

hw.pci.enable_msix: 0
hw.pci.enable_msi: 1

- Still using the Intel drivers

By default, using legacy IRQ or MSI means the cards/interfaces only use a single queue - MSI-X by default enables multiple queues ... this aspect could also be something to look into further.

[iMx]

6 days, no PPPoE drops/flaps, running with MSIX disabled and MSI enabled.

Probably after Christmas/NY, I will try re-enabling MSIX - to see if the various BIOS changes made have had any impact.

However, my hunch, is that running on MSI - i.e 1 queue per NIC - is potentially the likely cause/fix.  In which case, trying to disable MSIX on purely the PPPoE physical interface (the only interface I see issues with) might be the way forward.

A bit confused with the below, but something to consider regardless:

Code Select Expand

machdep.disable_msix_migration: Disable migration of MSI-X interrupts between CPUs


Wondering whether setting 'net.isr.bindthreads: 1' (as I have it at the moment and had with MSIX enabled originally) should not already be doing something similar to the former?!

Code Select Expand

net.isr.bindthreads: Bind netisr threads to CPUs.


...ISR being interrupt service routine.

Although, the former would presumably apply to all MSIX - the latter, just the network side of things?!

[iMx]

PPPoE has now been 'up' continuously for 700 hours, ~30 days.

Will re-enable MSIX within the next week, to see if problems re-occur - if not, possible some of the various BIOS settings noted above were the root cause.

[AllesMangoBrudi]

Thanks iMx for your thorough tests and explanations, you know what you're doing!

I have exactly a similar issue and only a reboot fixes it. I tried following your approach and wondered: How did you disable msi-x for the pppoe device? I understand you didn't disable it globally?

[iMx]

I haven't yet re-enabled MSIX - but I disabled it for everything, globally, as this was the only option when using the Intel-own drivers:

Code Select Expand

hw.pci.enable_msix: 0
hw.pci.enable_msi: 1

I wanted to 'prove' stability, which I seem to have now done. Current PPPoE uptime:

Code Select Expand

Uptime 1194:41:36

I just haven't gotten around to re-enabling MSIX yet, to see whether the various other BIOS changes I made, had any impact (ASPM, legacy USB, etc, etc).

If you run:

Code Select Expand

sysctl -A | grep msix

... you might be able to see if you can disable it for just the physical interface that PPPoE runs on.  I can't recall if the FreeBSD drivers had the ability to disable per interface, possibly varies depending on the NIC.

[elclay]

Finding the solution to this has been difficult, congratulations for being consistent, I abandoned it a long time ago. By the way, specifically how did you deactivate MSIX?

[phoenix]

The answer to your question is in the post above yours. ;)