Messages

sorry, i've reported it to bugzilla, just wanted to put it here in case it helped someone. apologies for the short story, not very good at doing stuff like this.

follow up to my own post. i think the flapping isnt the adapter at all, its a **locking bug in freebsds mii layer**. patch at the bottom

quick recap for anyone finding this fresh. tp-link ue306 (ax88179b) as wan on opnsense 26.7, vlan 10, spark fibre 930/500. stock axge did 57-73 mbps. i found four bugs in `if_axge.c` (rx aggregation timer way too short, two read helpers handing back uninitialised stack memory, `AXGE_PADDING` defined but never set, one rx transfer in flight) and that got it to **979 mbps**

but it still flapped. `ue0: link state changed` a few hundred times a minute, and the wan lease went with it every time. and it was weirdly all or nothing. it once ran 2 hours 18 minutes completely clean at 979, then i power cycled the box for an unrelated reason and it went straight back to flapping constantly. same kernel, same cable, same everything. that bistable thing is what made it so hard to pin down, i kept thinking id fixed it when id just got a good run

so heres what i think is happening. the mii layer keeps the current media (speed, duplex, link up or down) in a couple of shared variables, and it assumes that while its updating them its holding the driver lock so nothing else can look. thats fine on a normal pci nic, reading a phy register there is a register poke and a short poll, all of it under the lock, never sleeping

but on usb its different. every phy register read is a control transfer, and `usbd_do_request_flags()` **drops the mutex you hand it while it waits**. thats what passing it in is for. so the driver lock gets released and retaken on every single register read. `mii_tick()` (the once a second poll) and `mii_pollstat()` (anything asking the interface what its media is) end up running straight through each other, and one of them reads the media variables while the other is halfway through writing them

the code reads as correct, which is part of why it took me so long. `axge_ifmedia_sts()` does hold `AXGE_LOCK` across `mii_pollstat()`, and `axge_tick()` runs with it held. they still arent serialised, because the lock doesnt survive the transfer

went back through the logs and it was the media value that pointed somewhere. every DOWN event carried `0x20`, which is a bare `IFM_ETHER` with no speed and no duplex in it. the media word gets built in stages and `0x20` is what it holds for a few microseconds at the very start, before the speed goes in. `ukphy_status()` sets exactly that at the top, and every single exit path out of it fills in either a real speed or `IFM_NONE`, so a finished pass cant leave `0x20` behind. seeing it arrive looks like catching the other pass mid write

also every DOWN was followed by an UP microseconds later, and every one of those UPs said `0x100030`, which is 1000baseT full duplex. hundreds of events and it never once came back at 100baseTX or half duplex. that doesnt feel like a link genuinely renegotiating to me, though if theres a reason it would look like that id be glad to hear it

and it feeds itself, which id say is why it looked so random. a torn read makes `mii_linkchg()` fire off a link state change. whatever reacts to that goes and asks the interface what its media is, and that question is another mii pass, which tears again. so once it starts it doesnt stop on its own

stuff i chased that turned out to be nothing, in case it saves someone else the time:

- thought autoneg was restarting by itself, so i counted every BMCR write. two resets in an entire run. not it
- thought it was usb errors. tx errors 0, rx errors 0, the usb layer is completely healthy the whole time
- thought the driver was resetting the phy. init 1, mediaupd 1, reset 1, while statchg hit 659, so something was reacting to a link change the phy doesnt look like it made
- built a driver that lied and always reported link up in BMSR. that one was my favourite dead end, i was certain. changed nothing at all, which fits the corruption happening between the reads rather than in the bit
- thought it was load. it flaps at idle too, 426 events in 2.5 minutes at about 13 packets a second

the fix i went with is to make whole mii passes exclusive. a busy flag in the drivers per device struct, slept on with `msleep()` and the existing `sc_mtx`, wrapped around `mii_tick()`, `mii_pollstat()` and `mii_mediachg()`. i couldnt use a second mutex for this, the transfer sleeps and a normal mutex held across a sleep panics. the flag works where the lock doesnt, because it survives the lock getting dropped

40 second windows, before and after:

```
link state changes  186 / 313 / 139  ->  0 / 0 / 0
miibus_statchg      659              ->  4
phy register reads  57010            ->  4536
torn media words    170              ->  0
```

the 4 statchg calls left are just the transitions at startup, and it stays at 4. i also added a counter for how many passes had to wait on another one and it hit 254, then 538 on a longer run, so theyre landing on each other constantly, i was expecting a much narrower window than that. interface counters after: 0 input errors, 0 drops, 0 output errors across 1.89 GB in and 1.05 GB out

**979 down, 465 up, best single run 983** on a 930/500 plan. the 465 upload comes from a tx queue depth change (1 to 4) i havent submitted yet, without that expect about 250 up

i dont think this is an axge problem specifically. i grepped the tree and `if_axe`, `if_muge`, `if_smsc` and `if_ure` all call `mii_attach`, `mii_tick` and `mii_pollstat` exactly the same way axge does, and they all reach the phy over the same control transfers that drop the mutex. whether they actually hit it in practice i cant say, i only have the one adapter. so if youve got a usb nic on freebsd that flaps for no reason and youve just lived with it, might be worth a look. someone who actually knows the mii layer should probably say whether the proper fix belongs in there rather than being redone in every driver. i do wonder if some of usb ethernets reputation on freebsd comes from this, though thats one adapter and a lot of extrapolating

patch attached, 100 lines, touches `if_axge.c` and `if_axgereg.h`. applies on its own, no dependency on the earlier four. compiles clean against stable/26.7. going on the freebsd bug alongside this. its just the one module so applying it means building `if_axge.ko`, not a whole kernel

if youre on the cdce workaround (the usb quirk forcing config 1) none of this should touch you, youre stable at around 350/500 already. patched axge is doing 979/465 here against that, so its there if you fancy a swap. if anyone else with a usb nic gives it a go id be keen to hear how it went

thanks! yeah thats what made me go looking, the numbers just didnt add up for a superspeed link. hope it helps someone else with one of these!!

heads up before the technical bit, i dont really do forum posts, panic disorder, so this is a bit outside my comfort zone. but i spent an evening on this and it seems genuinely useful to anyone with one of these adapters, so here it is

setup: tp-link ue306 (ax88179b) as wan on opnsense 26.7, vlan 10 for spark fibre nz, 930/930 plan

out of the box axge attaches but is basically broken on the b revision. i was getting 57-73 mbps with input drops climbing constantly, and dhclient spamming send_packet: No buffer space available — half the time it couldnt even hold a lease. only workaround anyone documents is a usb quirk forcing config 1 so the generic cdce driver takes over instead. thats stable but caps at 350

went through the driver instead. same driver, now 979 mbps. four separate bugs, all in sys/dev/usb/net/if_axge.c

1. rx aggregation hold timer is 12x too short. axge_bulk_size[], first row, superspeed gigabit. its 0x004f. the chip never accumulates so it does one ethernet frame per usb transfer. 58k packets/sec, capped ~700mbps, router cpu 97 percent idle. swept it live during a transfer:


0x004f -> 705 Mbps, 58k pkt/s   (default)
0x0100 -> 952 Mbps, 79k pkt/s
0x0400 -> 979 Mbps, 81k pkt/s
0x4000 -> 979 Mbps, 81k pkt/s
went back to 0x004f at the end and it dropped again, so its the timer not something else. this one isnt b specific, it affects every axge adapter

2. axge_read_cmd_1/2 return uninitialised stack memory when the control transfer fails. they call axge_read_mem() and throw away the error. these are control transfers sharing the bus with bulk data and they absolutely do fail under load, so the mii layer gets garbage as link status. link flapped 462 times in 30 seconds under load, zero at idle. thats what was killing my dhcp lease every couple of minutes

3. AXGE_PADDING is defined in the header and never used anywhere. linux sets that bit when a frame plus header lands exactly on a usb packet boundary, freebsd doesnt, so the tx pipe wedges. this is where the ENOBUFS came from

4. only one rx transfer in flight. raised to 4, same as if_ure does. took packets/sec from 162 to 42k. tried 8 and it got much worse, 4 is right

single patch against stable/26.7, applies clean, builds with no warnings, running on my box right now. filing it on freebsd bugzilla too since thats where the driver actually lives — link once its up

tested on freebsd 15.1-p1 / opnsense 26.7, ax88179b, 1000baseT full duplex over superspeed, vlan tagged wan. before: 57-73 on axge, 350 via the cdce workaround. after: 979, no errors, no drops, no tcp retransmits

i only have the one adapter so i cant confirm on plain ax88179 or the a revision, but nothing in 1, 2 or 4 is revision specific. would be interested if anyone with other axge hardware tries the timer change, i suspect theres a lot of people on 500-700 who think thats just what usb ethernet does

patch is attached (axge-fixes.patch), applies with patch -p1 from the root of an opnsense/freebsd source tree.