Help please re Connection Issues after updating to Full Fibre with Open Reach UK

[Thorrrr]

ARP Table Memory Exhaustion & Connection Drops on ADSL and Full Fibre

The Issue
ADSL (75 Mbps):

Under load (e.g., downloading a 2GB file), I see ARP-related errors in the logs:
bash
Copy code
The command '/usr/sbin/arp -s '192.168.23.xxx' 'xx:xx:xx:xx:xx:xx'' returned exit code '1',
the output was 'arp: set 192.168.23.xxx: Cannot allocate memory'
Despite the relatively low bandwidth, connections appear stable most of the time but still occasionally drop.
Full Fibre:

When testing the same 2GB download over full fibre, the connection drops at random points (e.g., 10%, 44%).
Logs still show the same ARP memory exhaustion errors.
The ARP table on my network contains around 67 devices, including IoT devices, Nvidia devices, and standard clients.

Steps I've Taken So Far
1. System Tunables Adjustments
I've applied the following changes to improve resource allocation and ARP handling:

kern.ipc.nmbclusters = 262144
kern.ipc.maxsockbuf = 8388608
net.inet.ip.intr_queue_maxlen = 4096
net.link.bridge.pfil_bridge = 1
2. Clearing ARP Table
I attempted to clear the ARP table with:

bash
Copy code
arp -d -a
Result: Operation not permitted for most entries, suggesting they may be dynamically allocated or controlled by another process.
3. Diagnostics
ARP Table:
The table has 67 entries, which shouldn't overwhelm the system.
System Logs:
Repeated ARP-related errors and occasional memory exhaustion messages.
Firewall Live View:
Some state violations appear as "Default Deny" on WAN during downloads.
4. Interface and Hardware
MTU has been checked and is set to 1492 for PPPoE.
Disabled hardware offloading (checksum, TSO, LRO) under Interfaces > Settings.
5. Monitoring ARP Traffic
Using tcpdump to check ARP activity:

bash
Copy code
tcpdump -n -e -ttt -i re1 arp
No significant ARP flooding has been observed so far.
6. System Resources
Memory: No immediate memory pressure observed in top -aSH.
State table usage remains well within limits (799000 default).
My Network Setup
Firewall: OPNsense (latest version, fully updated).
WAN: PPPoE on ADSL and Full Fibre (IPoE setup).
LAN Interface: re1 with ~67 devices (IoT, Nvidia devices, Amazon products, and others).
Questions
Could this be caused by a misbehaving device or something stuck in the ARP table?
Are there other tunables I should tweak to further improve ARP handling or buffer limits?
Should I explore potential hardware limitations on my network card (Realtek NIC)?
Is there a better way to clear ARP or enforce limits for stale entries?
I'm open to any ideas, including further diagnostics, tunable adjustments, or hardware recommendations. I appreciate your time and suggestions!

[cookiemonster]

Please upload the full dmesg log from clean boot up to have another look but I imagine most errors are symptom of the problem with the realtek nic not being stable when stressed.

[Thorrrr]

Code Select Expand

dale@opnsense:~ $ dmesg
---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.1-RELEASE-p6 stable/24.7-n267939-fd5bc7f34e1 SMP amd64
FreeBSD clang version 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15a9eac9)
VT(vga): resolution 640x480
CPU: Intel(R) Celeron(R) CPU N3450 @ 1.10GHz (1094.40-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x506c9  Family=0x6  Model=0x5c  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2294e283<FSGSBASE,TSCADJ,SMEP,ERMS,NFPUSG,MPX,PQE,RDSEED,SMAP,CLFLUSHOPT,PROCTRACE,SHA>
  Structured Extended Features3=0x2c000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x79<RDCL_NO,SKIP_L1DFL_VME,SSB_NO,MDS_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8077860864 (7703 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL  EDK2    >
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119
Launching APs: 2 3 1
random: entropy device external interface
wlan: mac acl policy registered
kbd1 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 15.0.
smbios0: <System Management BIOS> at iomem 0xf05e0-0xf05fe
smbios0: Version: 3.0, BCD Revision: 3.0
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
acpi0: <ALASKA A M I >
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 8 on acpi0
Timecounter "HPET" frequency 19200000 Hz quality 950
Event timer "HPET" frequency 19200000 Hz quality 550
Event timer "HPET1" frequency 19200000 Hz quality 440
Event timer "HPET2" frequency 19200000 Hz quality 440
Event timer "HPET3" frequency 19200000 Hz quality 440
Event timer "HPET4" frequency 19200000 Hz quality 440
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0x90000000-0x90ffffff,0x80000000-0x8fffffff irq 19 at device 2.0 on pci0
vgapci0: Boot video device
hdac0: <Intel Broxton-P HDA Controller> mem 0x91310000-0x91313fff,0x91000000-0x910fffff irq 25 at device 14.0 on pci0
pci0: <simple comms> at device 15.0 (no driver attached)
pci0: <simple comms> at device 15.1 (no driver attached)
pci0: <simple comms> at device 15.2 (no driver attached)
ahci0: <Intel Apollo Lake AHCI SATA controller> port 0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0x91314000-0x91315fff,0x9131a000-0x9131a0ff,0x91319000-0x913197ff irq 19 at device 18.0 on pci0
ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pcib1: <ACPI PCI-PCI bridge> irq 22 at device 19.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 22 at device 20.0 on pci0
pci2: <ACPI PCI bus> on pcib2
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0x91204000-0x91204fff,0x91200000-0x91203fff irq 22 at device 0.0 on pci2
re0: Using 1 MSI-X message
re0: ASPM disabled
re0: Chip rev. 0x54000000
re0: MAC rev. 0x00100000
miibus0: <MII bus> on re0
rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 00:e0:4c:11:7e:78
re0: netmap queues/slots: TX 1/256, RX 1/256
pcib3: <ACPI PCI-PCI bridge> irq 23 at device 20.1 on pci0
pci3: <ACPI PCI bus> on pcib3
re1: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xd000-0xd0ff mem 0x91104000-0x91104fff,0x91100000-0x91103fff irq 23 at device 0.0 on pci3
re1: Using 1 MSI-X message
re1: ASPM disabled
re1: Chip rev. 0x54000000
re1: MAC rev. 0x00100000
miibus1: <MII bus> on re1
rgephy1: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus1
rgephy1:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re1: Using defaults for TSO: 65518/35/2048
re1: Ethernet address: 00:e0:4c:11:7e:79
re1: netmap queues/slots: TX 1/256, RX 1/256
xhci0: <Intel Apollo Lake USB 3.0 controller> mem 0x91300000-0x9130ffff irq 17 at device 21.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
sdhci_pci0: <Intel Apollo Lake eMMC 5.0 Controller> mem 0x91318000-0x91318fff,0x91317000-0x91317fff irq 39 at device 28.0 on pci0
sdhci_pci0: 1 slot(s) allocated
mmc0: <MMC/SD bus> on sdhci_pci0
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_button0: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff pnpid PNP0900 on isa0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounter "TSC" frequency 1094400261 Hz quality 1000
Timecounters tick every 1.000 msec
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
hdacc0: <Intel Broxton HDA CODEC> at cad 2 on hdac0
hdaa0: <Intel Broxton Audio Function Group> at nid 1 on hdacc0
pcm0: <Intel Broxton (HDMI/DP 8ch)> at nid 3 on hdaa0
mmcsd0: 31GB <MMCHC DA4032 0.1 SN E3F4C924 MFG 05/2022 by 69 0x0000> at mmc0 200.0MHz/8bit/8192-block
mmcsd0boot0: 4MB partition 1 at mmcsd0
mmcsd0boot1: 4MB partition 2 at mmcsd0
mmcsd0rpmb: 17MB partition 3 at mmcsd0
Trying to mount root from zfs:zroot/ROOT/default []...
uhub0: 15 ports with 15 removable, self powered
re0: link state changed to UP
re1: link state changed to UP
ichsmb0: <Intel Broxton SMBus controller> port 0xf040-0xf05f mem 0x91316000-0x913160ff irq 20 at device 31.1 on pci0
smbus0: <System Management Bus> on ichsmb0
lo0: link state changed to UP
pflog0: permanently promiscuous mode enabled
re0: link state changed to DOWN
vlan0: changing name to 'vlan01'
re1: link state changed to DOWN
ng0: changing name to 'pppoe0'
re0: link state changed to UP
vlan01: link state changed to UP
re1: link state changed to UP

[cookiemonster]

thanks. Up to this it seems all is good. Log ends in link state changed to UP for both interfaces. Upload again when with all messages after the troubles begin.

[Thorrrr]

HI cookiemonster

Sorry when i run it it just ends instantly am i missing something  ?

FYI i have tried again with a 2gb file in sabnzb and at 10 lost all connections seems to point to my Zimaboard router or my bad config :( 

Code Select Expand

dale@opnsense:~ $ dmesg
---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.1-RELEASE-p6 stable/24.7-n267981-8375762712f SMP amd64
FreeBSD clang version 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15a9eac9)
VT(vga): resolution 640x480
CPU: Intel(R) Celeron(R) CPU N3450 @ 1.10GHz (1094.40-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x506c9  Family=0x6  Model=0x5c  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2294e283<FSGSBASE,TSCADJ,SMEP,ERMS,NFPUSG,MPX,PQE,RDSEED,SMAP,CLFLUSHOPT,PROCTRACE,SHA>
  Structured Extended Features3=0x2c000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x79<RDCL_NO,SKIP_L1DFL_VME,SSB_NO,MDS_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8077860864 (7703 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL  EDK2    >
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119
Launching APs: 2 3 1
random: entropy device external interface
wlan: mac acl policy registered
kbd1 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 15.0.
smbios0: <System Management BIOS> at iomem 0xf05e0-0xf05fe
smbios0: Version: 3.0, BCD Revision: 3.0
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
acpi0: <ALASKA A M I >
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 8 on acpi0
Timecounter "HPET" frequency 19200000 Hz quality 950
Event timer "HPET" frequency 19200000 Hz quality 550
Event timer "HPET1" frequency 19200000 Hz quality 440
Event timer "HPET2" frequency 19200000 Hz quality 440
Event timer "HPET3" frequency 19200000 Hz quality 440
Event timer "HPET4" frequency 19200000 Hz quality 440
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0x90000000-0x90ffffff,0x80000000-0x8fffffff irq 19 at device 2.0 on pci0
vgapci0: Boot video device
hdac0: <Intel Broxton-P HDA Controller> mem 0x91310000-0x91313fff,0x91000000-0x910fffff irq 25 at device 14.0 on pci0
pci0: <simple comms> at device 15.0 (no driver attached)
pci0: <simple comms> at device 15.1 (no driver attached)
pci0: <simple comms> at device 15.2 (no driver attached)
ahci0: <Intel Apollo Lake AHCI SATA controller> port 0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0x91314000-0x91315fff,0x9131a000-0x9131a0ff,0x91319000-0x913197ff irq 19 at device 18.0 on pci0
ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pcib1: <ACPI PCI-PCI bridge> irq 22 at device 19.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 22 at device 20.0 on pci0
pci2: <ACPI PCI bus> on pcib2
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0x91204000-0x91204fff,0x91200000-0x91203fff irq 22 at device 0.0 on pci2
re0: ASPM disabled
re0: Chip rev. 0x54000000
re0: MAC rev. 0x00100000
miibus0: <MII bus> on re0
rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 00:e0:4c:11:7e:78
re0: netmap queues/slots: TX 1/256, RX 1/256
pcib3: <ACPI PCI-PCI bridge> irq 23 at device 20.1 on pci0
pci3: <ACPI PCI bus> on pcib3
re1: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xd000-0xd0ff mem 0x91104000-0x91104fff,0x91100000-0x91103fff irq 23 at device 0.0 on pci3
re1: ASPM disabled
re1: Chip rev. 0x54000000
re1: MAC rev. 0x00100000
miibus1: <MII bus> on re1
rgephy1: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus1
rgephy1:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re1: Using defaults for TSO: 65518/35/2048
re1: Ethernet address: 00:e0:4c:11:7e:79
re1: netmap queues/slots: TX 1/256, RX 1/256
xhci0: <Intel Apollo Lake USB 3.0 controller> mem 0x91300000-0x9130ffff irq 17 at device 21.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
sdhci_pci0: <Intel Apollo Lake eMMC 5.0 Controller> mem 0x91318000-0x91318fff,0x91317000-0x91317fff irq 39 at device 28.0 on pci0
sdhci_pci0: 1 slot(s) allocated
mmc0: <MMC/SD bus> on sdhci_pci0
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_button0: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff pnpid PNP0900 on isa0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounter "TSC" frequency 1094400149 Hz quality 1000
Timecounters tick every 1.000 msec
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
hdacc0: <Intel Broxton HDA CODEC> at cad 2 on hdac0
hdaa0: <Intel Broxton Audio Function Group> at nid 1 on hdacc0
pcm0: <Intel Broxton (HDMI/DP 8ch)> at nid 3 on hdaa0
mmcsd0: 31GB <MMCHC DA4032 0.1 SN E3F4C924 MFG 05/2022 by 69 0x0000> at mmc0 200.0MHz/8bit/8192-block
mmcsd0boot0: 4MB partition 1 at mmcsd0
mmcsd0boot1: 4MB partition 2 at mmcsd0
mmcsd0rpmb: 17MB partition 3 at mmcsd0
Trying to mount root from zfs:zroot/ROOT/default []...
uhub0: 15 ports with 15 removable, self powered
re0: link state changed to UP
re1: link state changed to UP
ichsmb0: <Intel Broxton SMBus controller> port 0xf040-0xf05f mem 0x91316000-0x913160ff irq 20 at device 31.1 on pci0
smbus0: <System Management Bus> on ichsmb0
lo0: link state changed to UP
pflog0: permanently promiscuous mode enabled
re0: link state changed to DOWN
vlan0: changing name to 'vlan01'
re1: link state changed to DOWN
ng0: changing name to 'pppoe0'
re0: link state changed to UP
vlan01: link state changed to UP
re1: link state changed to UP
dale@opnsense:~ $

[jim1985]

Try limiting SABNZB to use less bandwidth during download.

Yes this isn't ideal, but it could serve as an indication as to whether it is your current hardware than is struggling to cope with the increased bandwidth now available to you since upgrading to Full Fibre.


FWIW, I'm in the UK on a 500Mbit FTTP connection, running as PPPoE through OPNSense and have no trouble going flat out with SABNZB, I am however using different hardware for my OPNSense installation, it's running on an HP T730 Thin Client that I have thrown a quad port Intel I350 card in to

[cookiemonster]

quite clear:
lo0: link state changed to UP
pflog0: permanently promiscuous mode enabled

so far so good, interface is up. Then:

re0: link state changed to DOWN

link (NIC) goes down and other nic goes down too:
re1: link state changed to DOWN
ng0: changing name to 'pppoe0' <- can ignore

nic back up:
re0: link state changed to UP
vlan01: link state changed to UP
re1: link state changed to UP

So this is the usual symnptom when we've seen being called "interface is flapping". As to why, as said before Realtek nics are rubbish under stress with them not providing good drivers for freeBSD.

What to do: try with another device/NIC. I expect all will be good and Chirstmas will be less spoiled.

[Thorrrr]

quite clear:
lo0: link state changed to UP
pflog0: permanently promiscuous mode enabled

so far so good, interface is up. Then:

re0: link state changed to DOWN

link (NIC) goes down and other nic goes down too:
re1: link state changed to DOWN
ng0: changing name to 'pppoe0' <- can ignore

nic back up:
re0: link state changed to UP
vlan01: link state changed to UP
re1: link state changed to UP

So this is the usual symnptom when we've seen being called "interface is flapping". As to why, as said before Realtek nics are rubbish under stress with them not providing good drivers for freeBSD.

What to do: try with another device/NIC. I expect all will be good and Chirstmas will be less spoiled.

Do you have any recommendations re best device/NIC to buy without breaking the bank but future proofing as much as you can ?

[cookiemonster]

Try installing the vendor's driver on your OPN first. It might help and save you some dosh.
Intel NICs are the usual recommendation. Second hand from ebay, go for two ports ones like link work well. Newer ones from the i210 and similar are a bit hit and miss depending on chipset revision. You need to research a little. Some info in this forum.
For the driver, search on your OPN in System | Plugins for os-realtek-re. Needs reboot.

[Thorrrr]

OK
I added the drivers today as suggested the "os-realtek-re" been running all day fingers crossed looks good.
Tried downloading some big files through the router to stress it looks OK so far.
Cheers for all the help and advise great forums and members

[cookiemonster]

fingers crossed.

[Thorrrr]

OK
I have had no crashing of loss of signal yet. But i have done some testing today directly with a wired connection directly on the modem.
I have tried with a laptop with a USB 3.1 connection direct connect with my 900mb the best i can get is about 456mb i have tried about 6 different servers locations but never pass 500b mb.

Id i attach the Plusnet modem i get 896 mb instantly and 160mb upload  which is what i was expecting . So everything points to my Opnsense router but not sure how to check where the bottleneck is!

I have ordered Dell INTEL I350-T4 Low Profile PCI-E 4-port Network Card PN: K9CR1 to attache to the PCI on my Zimaboard. Not 100% sure but from reading this should be better as a NIC compatibility.

ANy thought on how i can check what is throttling or bottle necking the sped ?

[cookiemonster]

PPoE connection on WAN, right? If so, then sorry, I have no advice to give on getting such a connection to reach gigabit.
I'm sure someone else will be able to give you some hints on what to try.

[jim1985]

OK
I have had no crashing of loss of signal yet. But i have done some testing today directly with a wired connection directly on the modem.
I have tried with a laptop with a USB 3.1 connection direct connect with my 900mb the best i can get is about 456mb i have tried about 6 different servers locations but never pass 500b mb.

Id i attach the Plusnet modem i get 896 mb instantly and 160mb upload  which is what i was expecting . So everything points to my Opnsense router but not sure how to check where the bottleneck is!

I have ordered Dell INTEL I350-T4 Low Profile PCI-E 4-port Network Card PN: K9CR1 to attache to the PCI on my Zimaboard. Not 100% sure but from reading this should be better as a NIC compatibility.

ANy thought on how i can check what is throttling or bottle necking the sped ?

What other services (if any at all) do you have running on your OPNSense installation?
I noticed that your Zimaboard uses a fairly weak Celeron processor. It's possible perhaps that that is where your bottleneck is coming from

[cookiemonster]

Try a google search for "pppoe single threaded" and you should get some more good advice. Some of will circle you back to this forum.