Kernal Panic - WPA broken for wifi after 24.7 update

[fuskadoo]

Hardware: Qotom-Q355G4

After updating to 24.7, Ralink wifi gets broken and I cannot get it working again.  I use the built in basic wifi just as an isolated IoT for a few basic risky devices, so it's important.  Also I don't even see the SSID being broadcast anymore, so it has me wondering if it's loading properly.  It's there an look correct in Interfaces > Wiresless > Devices.

In the wireless logs I can see that right after upgrade it changes to "WPA rekeying GTK" repeating instead of doing proper handshakes.

Code Select Expand


# cat /var/log/wireless/latest.log

<31>1 2024-07-31T16:43:15-05:00 home.arpa hostapd 7345 - [meta sequenceId="23"] run0_wlan1: STA 22:6a:10:ba:15:73 WPA: received EAPOL-Key frame (2/2 Group)
<30>1 2024-07-31T16:43:15-05:00 home.arpa hostapd 7345 - [meta sequenceId="24"] run0_wlan1: STA 22:6a:10:ba:15:73 WPA: group key handshake completed (RSN)
<31>1 2024-07-31T16:43:15-05:00 home.arpa hostapd 7345 - [meta sequenceId="25"] run0_wlan1: STA 4c:a1:61:04:d8:27 WPA: received EAPOL-Key frame (2/2 Group)
<30>1 2024-07-31T16:43:15-05:00 home.arpa hostapd 7345 - [meta sequenceId="26"] run0_wlan1: STA 4c:a1:61:04:d8:27 WPA: group key handshake completed (RSN)
<31>1 2024-07-31T16:43:15-05:00 home.arpa hostapd 7345 - [meta sequenceId="27"] run0_wlan1: STA 22:6a:10:ba:14:e4 WPA: received EAPOL-Key frame (2/2 Group)
<30>1 2024-07-31T16:43:15-05:00 home.arpa hostapd 7345 - [meta sequenceId="28"] run0_wlan1: STA 22:6a:10:ba:14:e4 WPA: group key handshake completed (RSN)
<31>1 2024-07-31T16:43:15-05:00 home.arpa hostapd 7345 - [meta sequenceId="29"] run0_wlan1: STA 22:6a:10:ba:15:67 WPA: received EAPOL-Key frame (2/2 Group)
<30>1 2024-07-31T16:43:15-05:00 home.arpa hostapd 7345 - [meta sequenceId="30"] run0_wlan1: STA 22:6a:10:ba:15:67 WPA: group key handshake completed (RSN)
<31>1 2024-07-31T16:43:16-05:00 home.arpa hostapd 7345 - [meta sequenceId="31"] run0_wlan1: STA 33:61:32:3a:52:1c WPA: EAPOL-Key timeout
<31>1 2024-07-31T16:43:16-05:00 home.arpa hostapd 7345 - [meta sequenceId="32"] run0_wlan1: STA 33:61:32:3a:52:1c WPA: sending 1/2 msg of Group Key Handshake
<31>1 2024-07-31T16:43:16-05:00 home.arpa hostapd 7345 - [meta sequenceId="33"] run0_wlan1: STA 33:61:32:3a:52:1c WPA: received EAPOL-Key frame (2/2 Group)
<30>1 2024-07-31T16:43:16-05:00 home.arpa hostapd 7345 - [meta sequenceId="34"] run0_wlan1: STA 33:61:32:3a:52:1c WPA: group key handshake completed (RSN)
<31>1 2024-07-31T16:50:10-05:00 home.arpa hostapd 48511 - [meta sequenceId="1"] run0_wlan1: WPA rekeying GTK
<31>1 2024-07-31T16:51:11-05:00 home.arpa hostapd 48511 - [meta sequenceId="1"] run0_wlan1: WPA rekeying GTK
<31>1 2024-07-31T16:52:11-05:00 home.arpa hostapd 48511 - [meta sequenceId="2"] run0_wlan1: WPA rekeying GTK
<31>1 2024-07-31T16:53:10-05:00 home.arpa hostapd 48511 - [meta sequenceId="1"] run0_wlan1: WPA rekeying GTK
<31>1 2024-07-31T16:55:34-05:00 home.arpa hostapd 46747 - [meta sequenceId="1"] run0_wlan1: WPA rekeying GTK


# dmesg | grep run0

Code Select Expand


run0 on uhub1
run0: <Ralink 802.11 n WLAN, class 0/0, rev 2.00/1.01, addr 1> on usbus0
run0: MAC/BBP RT5390 (rev 0x0503), RF RT5370 (MIMO 1T1R), address 24:0a:64:a0:fc:34
run0: [HT] Enabling 802.11n
wlan0: changing name to 'run0_wlan1'
run0: firmware RT3071 ver. 0.33 loaded
run0: firmware RT3071 ver. 0.33 loaded
run0: firmware RT3071 ver. 0.33 loaded


Code Select Expand

#  sysctl -n net.wlan.devices
run0


Anyone know how to troubleshoot this issue with 24.7?

[fuskadoo]

What I have figured out is there is something really broken with WPA.  This has always worked beautifully until I got to 24.7.

What I have found so far:
1) Remove the wireless device all together, and remove the interface (setup over from scratch)
2) Reboot and then add it back in under INTERFACE > WIRELESS > DEVICES (listed as run0)
3) When I look under WIRELESS STATUS and it's working!  It can scan and see other access points!
4) Then I set the wireless device to ACCESS POINT and leave as authentication OPEN.  It works!  I can connect to the AP.
5) Now here is where it breaks!  When I enable WPA by clicking the box, the access point disappears and ceases to work going forward

6) Kernel panic!  If I try to go back to open authentication by unchecking WPA, it kernel craps and hard reboots the device.  It shouldn't kernel panic regardless, but something with WPA in 24.7 is really amiss. Picture attached.

I have submitted crash logs through the GUI.  Please let me know how I can troubleshoot this to help get a fix.

Thank you.

[franco]

We got a kernel panic reported for run(4) already. The suspicion is that the whole ieee80211 subsystem suffers from locking changes carried out in FreeBSD 14.1.


Cheers,
Franco

[fuskadoo]

Hi Franco,

I just saw this a few minutes ago which is a duplicate that I didn't initially find in my search. 

https://forum.opnsense.org/index.php?topic=41870.0

It is quite strange that with OPEN auth it seems to function fine.  But the second I enable WPA it fails and gets hung until I completely remove the wifi device and interface. It looks like it's well noted and hopefully these tips can help solving it at some point.

Take care.

[franco]

Yes, that's the one. I'm on it, but I have a feeling it will be a slow grind. Thanks for your input so far!


Cheers,
Franco

[jhelmly]

I am supporting a client using a 4-port FW box with a builtin Liteon wifi module.  I upgraded their appliance today from 24.1.10 to 24.7.2.  About 30 minutes later, I start getting complaints that the wifi is not working.  Even though everything seemed to be working (I even still had dhcp leases for the wifi interface), when I attempted to join something to the wifi, the SSID is not being broadcast.  The GUI and the CLI show the interface up, but it is not working.  I tried the kern.smb.disabled==1 setting and a reboot, but still the wifi is not working.  I also get many "WPA rekeying GTK" messages in the wireless log.

Are there any other options to try?  Is there a process to downgrade to 24.1.0 or do I need to rebuild and restore the configuration?

[doktornotor]

Are there any other options to try?

Other options to try - buying a proper access point and stop trying to use FreeBSD as one. I seriously don't know why people keep doing this to themselves despite the endless warnings.

[Kelutrel]

Are there any other options to try?

I agree with everyone here that FreeBSD should not be used as WiFi AP. However for logistics reasons I have some advantage in having a single piece of hardware working as Firewall/Switch/AP in my homespace, and up to now an OPNSense mini-PC has worked wonders for my needs.

I was able to make my "Ralink 802.11 n/g/b Wireless LAN USB Mini-Card" work in OPNSense 24.7.1+ by disabling both WEP/WPA and keeping the WiFi SSID hidden and somewhat randomly generated. Obviously the client devices must support WiFi Access Points with a hidden SSID.

There is an obvious security hole in having no WEP/WPA auth, that I partially covered by having the SSID randomized and hidden and allowing the DHCP to only accept known clients, so I would not suggest this in a professional office environment. But for my home network this config worked with my WiFi hardware, and allowed me to quickly restore the WiFi connectivity, so I thought to mention it here.

[siga75]

Hi,

just upgraded and facing the same issue. Any news?

Code Select Expand

root@myfw:~ # dmesg | grep run0
run0 on uhub0
run0: <Ralink 802.11 n WLAN, class 0/0, rev 2.00/1.01, addr 1> on usbus0
run0: MAC/BBP RT3070 (rev 0x0200), RF RT3020 (MIMO 1T1R), address 00:22:43:73:89:17
run0: [HT] Enabling 802.11n
wlan0: changing name to 'run0_wlan1'
run0: firmware RT2870 ver. 0.33 loaded
run0: firmware RT2870 ver. 0.33 loaded
run0: firmware RT2870 ver. 0.33 loaded
root@myfw:~ # tail /var/log/wireless/latest.log
<31>1 2024-10-15T17:07:21+02:00 myfw.signorini.in hostapd 17447 - [meta sequenceId="1"] run0_wlan1: WPA rekeying GTK
<31>1 2024-10-15T17:08:21+02:00 myfw.signorini.in hostapd 17447 - [meta sequenceId="1"] run0_wlan1: WPA rekeying GTK
<31>1 2024-10-15T17:09:21+02:00 myfw.signorini.in hostapd 17447 - [meta sequenceId="1"] run0_wlan1: WPA rekeying GTK
<31>1 2024-10-15T17:10:21+02:00 myfw.signorini.in hostapd 17447 - [meta sequenceId="1"] run0_wlan1: WPA rekeying GTK
<31>1 2024-10-15T17:11:21+02:00 myfw.signorini.in hostapd 17447 - [meta sequenceId="1"] run0_wlan1: WPA rekeying GTK
<31>1 2024-10-15T17:12:21+02:00 myfw.signorini.in hostapd 17447 - [meta sequenceId="1"] run0_wlan1: WPA rekeying GTK
<31>1 2024-10-15T17:13:21+02:00 myfw.signorini.in hostapd 17447 - [meta sequenceId="1"] run0_wlan1: WPA rekeying GTK
<31>1 2024-10-15T17:14:21+02:00 myfw.signorini.in hostapd 17447 - [meta sequenceId="1"] run0_wlan1: WPA rekeying GTK
<31>1 2024-10-15T17:15:21+02:00 myfw.signorini.in hostapd 17447 - [meta sequenceId="1"] run0_wlan1: WPA rekeying GTK
<31>1 2024-10-15T17:16:21+02:00 myfw.signorini.in hostapd 17447 - [meta sequenceId="2"] run0_wlan1: WPA rekeying GTK


[astronaut]

A bit late to the show, but with regards to the discussion part on using one device as a FW/AP, I use virtualization (Xen) with PCI passthrough for this. The hardware is an APU4D4. The host (Dom0 in Xen speak) is running vanilla Debian. The WiFi card is passed through to a VM (DomU) running OpenWRT, the OPNsense firewall is running in a seperate VM. Both are connected via a software bridge.

There are drawbacks to this solution, i. e. added complexity at a single point of failure or an (theoretically?) increased security risk, but in my case, it works quite well.

[siga75]

nevermind, just ordered a cheap AP, too bad, it was nice not to have an extra device

[guttermonk]

I'm in the same boat. I've started a new issue: https://github.com/opnsense/core/issues/8080

Can anyone recommend an inexpensive access point that won't add a terrible number of cords/cables to my rat's nest of wires?

[Patrick M. Hausen]

Can anyone recommend an inexpensive access point that won't add a terrible number of cords/cables to my rat's nest of wires?

I have become quite fond of Mikrotik. Pick according to budget, they have quite some choice.

[astronaut]

Can anyone recommend an inexpensive access point that won't add a terrible number of cords/cables to my rat's nest of wires?

At home, I am using Zyxel hardware for switches/APs, because I can easily install OpenWRT on those boxes due to their freely accessible serial port. By using OpenWRT, I don't need to get accustomed to a new user interface...  ;) I guess Zyxel's reputation is not the best, but the devices are doing a good job in my case.

As AP, I am using the Zyxel NWA 50AX.

PS: I just noticed that there are OpenWRT images available for some mikrotik devices as well.