OPNsense Forum

English Forums => 24.7, 24.10 Series => Topic started by: adamrc on July 25, 2024, 06:11:29 PM

Title: [SOLVED]: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: adamrc on July 25, 2024, 06:11:29 PM
This morning I upgraded from 24.1.10 to 24.7.  I've been successfully using the AT&T bypass which uses WPA_Supplicant along with Netgraph as seen in the instructions here:  https://github.com/owenthewizard/opnatt

Unfortunately, it hangs upon booting up.  See attached screenshot.  It no longer seems to pull the WAN IP anymore.  I can do a Control + C to continue to boot but even within the GUI, it doesn't pull an IP anymore.

Any ideas on what I could try to fix it?  It's been working great for so long prior to this update. 
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: PrototypeActual on July 26, 2024, 06:05:22 PM
I too have AT&T fiber but wasn't aware/using this bypass and was having issues with my AT&T and my backup Spectrum connections working. While rebooting I noticed it finally was allowing an internet connection but nothing worked after the reboot. Were you able to find a way around it or reverted back to 24.1.10 like me?

P.S.
Was setting up the bypass difficult for you or noticed any significant changes?
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: adamrc on July 26, 2024, 06:36:08 PM
The only fix I wanted to mess with at the time was to bring my AT&T gateway back into the mix and use that instead.  I'm curious as to if any others that have been using this AT&T gateway bypass have also experinced this issue after upgrading to 24.7.  I am really enjoying the much faster wireguard speeds on 24.7.

Setting up the bypass certainly took a bit of work but after it was all done, it has been super reliable with rock solid stability. I certainly want to get it back up and running as I don't like going through the AT&T gateway. This update is the first time any update has caused any type of disruption to my network. 
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: Ben S on July 27, 2024, 12:57:45 AM
I have no idea about AT&T or this bypass procedure but I had a quick read of the code in the repo you linked, and one thing I notice is your screenshot says 'Starting pfatt' but the latest code says 'Starting opnatt'.  It looks like that 'pfatt' message dates back to this commit - https://github.com/owenthewizard/opnatt/commit/134dd592d5ee459b88d94dac8a6110265ebba1a2 - in which many other things were changed, including how it runs wpa_supplicant.  I also notice your screenshot, near the bottom, it says 'wpa_supplicant running on PID', and doesn't show a PID number, suggesting wpa_supplicant is not running, which makes me think it's not calling wpa_supplicant in the right way perhaps.

So tl;dr - have you tried the latest upstream version of this script?

Also I wonder if you can find any more clues in any logs and/or try running the script from the console/SSH, to see if it gives any more diagnostics about what's going wrong.
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: adamrc on July 27, 2024, 03:24:19 AM
Thanks Ben S for the info.  I have not tried the latest code.  Will give that a try over the next couple of days and see what happens. 
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: jay4 on July 28, 2024, 07:09:11 PM
I don't know why I never learn. It was a bear to setup but I got it working seamlessly, I saw the shiny upgrade button and now my bypass isn't working. I was doing the bypass with the certs, no need for the att rg.I have yet to troubleshoot with tcpdump but I'm willing to bet it's the same issue with vlan0 not being handled correctly.
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: jeholliday on August 06, 2024, 08:59:17 PM
This is the last time I start a major OS upgrade right before going to sleep. I was in the same boat that I've never had any other problems with updates, but this one broke my ATT bypass. I went back to the gateway until I could sort it out and was reminded why the bypass solution was so much better. I've been experiencing random connection dropouts and can't get IPv6 to work in IP pass through mode.

However, I believe I have found the solution! I ran wpa_supplicant in verbose mode and got some warnings related to TLS.
EAP: Received EAP-Request id=29 method=13 vendor=0 vendorMethod=0
EAP: EAP entering state GET_METHOD
ngeth0: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=13
EAP: Status notification: accept proposed method (param=TLS)
EAP: Initialize selected EAP method: vendor 0 method 13 (TLS)
TLS: using phase1 config options
TLS: Trusted root certificate(s) loaded
OpenSSL: tls_connection_client_cert - SSL_use_certificate_file failed error:0A00018E:SSL routines::ca md too weak
TLS: Failed to set TLS connection parameters
ENGINE: engine deinit
EAP-TLS: Failed to initialize SSL.


It seems like we must have gotten a OpenSSL update or a wpa_supplicant update that makes it now reject the certificates as insecure by default. My solution is to update the wpa_supplicant.conf to include openssl_ciphers=DEFAULT@SECLEVEL=0. That makes it get further in the process. It then says that it also needs allow_unsafe_renegotiation=1 in the phase1 config. With both of those changes, I am now back in business. My complete wpa_supplicant.conf looks like this:

eapol_version=1
ap_scan=0
fast_reauth=1
openssl_ciphers=DEFAULT@SECLEVEL=0
network={
        eap=TLS
        eapol_flags=0
        key_mgmt=IEEE8021X
        phase1="allow_canned_success=1 allow_unsafe_renegotiation=1"
}


I don't know if this is the best solution, but it is working for now.
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: adamrc on August 06, 2024, 09:18:08 PM
I'm back up and running as well.  I scrapped everything and went to the 8311 Discord channel for bypassing and got the proper info.  Here are the files and contents that I used.  Netgraph is no longer used/needed anymore.

/usr/local/etc/rc.syshook.d/early/04-wpa (make sure to chmod +x this file)

#!/bin/sh
env OPENSSL_CONF=/conf/wpa/openssl.conf /usr/local/sbin/wpa_supplicant -Dwired -i igb0 -B -C /var/run/wpa_supplicant -c /conf/wpa/wpa_supplicant.conf


/conf/wpa/openssl.conf

openssl_conf = openssl_init

[openssl_init]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Options = UnsafeLegacyRenegotiation
MinProtocol = TLSv1
CipherString = DEFAULT@SECLEVEL=0


/conf/wpa/wpa_supplicant.conf

# Generated by 802.1x Credential Extraction Tool
# Copyright (c) 2018-2019 devicelocksmith.com
# Version: 1.04 windows 386
#
# Change file names to absolute paths
ctrl_interface=DIR=/var/run/wpa_supplicant
openssl_ciphers=DEFAULT@SECLEVEL=0
eapol_version=2
ap_scan=0
fast_reauth=1
network={
        ca_cert="/conf/wpa/ca.pem"
        client_cert="/conf/wpa/client.pem"
        eap=TLS
        eapol_flags=0
        identity="REDACTED" # Internet (ONT) interface MAC address must match this value
        key_mgmt=IEEE8021X
        phase1="allow_canned_success=1"
        private_key="/conf/wpa/private.pem"
}
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: effex on August 09, 2024, 11:28:24 PM
Mine is still not working with the code above. The openatt.sh script seems to work just fine, however, something is not working with wpa_supplicant. I tried manually executing the script in early and it gets to "daemonize" and gives me a permission denied. Any way to further debug it?
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: adamrc on August 09, 2024, 11:54:38 PM
Quote from: effex on August 09, 2024, 11:28:24 PM
Mine is still not working with the code above. The openatt.sh script seems to work just fine, however, something is not working with wpa_supplicant. I tried manually executing the script in early and it gets to "daemonize" and gives me a permission denied. Any way to further debug it?

No need for the openatt.sh script using my method above. Just ensure certs are in the folders specified in the file.
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: shermdog on August 11, 2024, 08:47:41 PM
Big shout out to the folks that organized this info.  Super clean implementation and working great on 24.1 as well.


Quote from: adamrc on August 06, 2024, 09:18:08 PM
I'm back up and running as well.  I scrapped everything and went to the 8311 Discord channel for bypassing and got the proper info.  Here are the files and contents that I used.  Netgraph is no longer used/needed anymore.

/usr/local/etc/rc.syshook.d/early/04-wpa (make sure to chmod +x this file)

#!/bin/sh
env OPENSSL_CONF=/conf/wpa/openssl.conf /usr/local/sbin/wpa_supplicant -Dwired -i igb0 -B -C /var/run/wpa_supplicant -c /conf/wpa/wpa_supplicant.conf


/conf/wpa/openssl.conf

openssl_conf = openssl_init

[openssl_init]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Options = UnsafeLegacyRenegotiation
MinProtocol = TLSv1
CipherString = DEFAULT@SECLEVEL=0


/conf/wpa/wpa_supplicant.conf

# Generated by 802.1x Credential Extraction Tool
# Copyright (c) 2018-2019 devicelocksmith.com
# Version: 1.04 windows 386
#
# Change file names to absolute paths
ctrl_interface=DIR=/var/run/wpa_supplicant
openssl_ciphers=DEFAULT@SECLEVEL=0
eapol_version=2
ap_scan=0
fast_reauth=1
network={
        ca_cert="/conf/wpa/ca.pem"
        client_cert="/conf/wpa/client.pem"
        eap=TLS
        eapol_flags=0
        identity="REDACTED" # Internet (ONT) interface MAC address must match this value
        key_mgmt=IEEE8021X
        phase1="allow_canned_success=1"
        private_key="/conf/wpa/private.pem"
}

Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: franco on August 11, 2024, 08:49:37 PM
If someone wants this integrated in the GUI I'd appreciate a GitHub ticket more than these discussions after random releases breaking this stuff we never had integrated. ;)


Cheers,
Franco
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: effex on August 19, 2024, 12:16:48 AM
Quote from: adamrc on August 09, 2024, 11:54:38 PM
Quote from: effex on August 09, 2024, 11:28:24 PM
Mine is still not working with the code above. The openatt.sh script seems to work just fine, however, something is not working with wpa_supplicant. I tried manually executing the script in early and it gets to "daemonize" and gives me a permission denied. Any way to further debug it?

No need for the openatt.sh script using my method above. Just ensure certs are in the folders specified in the file.


On the identity piece, is that the RG modem's former MAC address like the same as the opnaatt.sh script or are you saying my WAN mac address on my opnsense box?
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: effex on August 19, 2024, 12:48:54 AM
Quote from: franco on August 11, 2024, 08:49:37 PM
If someone wants this integrated in the GUI I'd appreciate a GitHub ticket more than these discussions after random releases breaking this stuff we never had integrated. ;)


Cheers,
Franco

Do I just open a request to have this functionality added? I'm happy to do so if that helps.
Title: Re: [SOLVED]: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: franco on August 19, 2024, 01:43:18 PM
Yep.
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: adamrc on August 19, 2024, 03:25:39 PM
Quote from: effex on August 19, 2024, 12:16:48 AM
Quote from: adamrc on August 09, 2024, 11:54:38 PM
Quote from: effex on August 09, 2024, 11:28:24 PM
Mine is still not working with the code above. The openatt.sh script seems to work just fine, however, something is not working with wpa_supplicant. I tried manually executing the script in early and it gets to "daemonize" and gives me a permission denied. Any way to further debug it?

No need for the openatt.sh script using my method above. Just ensure certs are in the folders specified in the file.


On the identity piece, is that the RG modem's former MAC address like the same as the opnaatt.sh script or are you saying my WAN mac address on my opnsense box?

It's the AT&T RG modem MAC address.
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: GPz1100 on August 24, 2024, 07:00:03 AM
Quote from: franco on August 11, 2024, 08:49:37 PM
,
Franco

Congrats on finally implementing the vlan 0 patches for wpa_supplicant. Why did it take so long?  Pfsense had it back in june of 2023.
Title: Re: [SOLVED]: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: franco on August 24, 2024, 07:01:22 AM
Quote from: GPz1100 on August 24, 2024, 07:00:03 AM
Congrats on finally implementing the vlan 0 patches for wpa_supplicant. Why did it take so long?  Pfsense had it back in june of 2023.

Do you want to be set to read only mode for trolling? My first and last warning.
Title: Re: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: selcouth on August 31, 2024, 11:25:06 PM
Quote from: adamrc on August 06, 2024, 09:18:08 PM
I'm back up and running as well.  I scrapped everything and went to the 8311 Discord channel for bypassing and got the proper info.  Here are the files and contents that I used.  Netgraph is no longer used/needed anymore.

/usr/local/etc/rc.syshook.d/early/04-wpa (make sure to chmod +x this file)

#!/bin/sh
env OPENSSL_CONF=/conf/wpa/openssl.conf /usr/local/sbin/wpa_supplicant -Dwired -i igb0 -B -C /var/run/wpa_supplicant -c /conf/wpa/wpa_supplicant.conf


/conf/wpa/openssl.conf

openssl_conf = openssl_init

[openssl_init]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Options = UnsafeLegacyRenegotiation
MinProtocol = TLSv1
CipherString = DEFAULT@SECLEVEL=0


/conf/wpa/wpa_supplicant.conf

# Generated by 802.1x Credential Extraction Tool
# Copyright (c) 2018-2019 devicelocksmith.com
# Version: 1.04 windows 386
#
# Change file names to absolute paths
ctrl_interface=DIR=/var/run/wpa_supplicant
openssl_ciphers=DEFAULT@SECLEVEL=0
eapol_version=2
ap_scan=0
fast_reauth=1
network={
        ca_cert="/conf/wpa/ca.pem"
        client_cert="/conf/wpa/client.pem"
        eap=TLS
        eapol_flags=0
        identity="REDACTED" # Internet (ONT) interface MAC address must match this value
        key_mgmt=IEEE8021X
        phase1="allow_canned_success=1"
        private_key="/conf/wpa/private.pem"
}


Thank you for posting this. I just moved to OPNsense from pFsense and was struggling with the ngeth0 method. The method you referenced worked perfectly.
Title: Re: [SOLVED]: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: Quasmo on September 11, 2024, 08:33:46 PM
Not sure what is causing the problem on my end...
I just switched from pfSense to OPNsense. I have setup the files similar to those set out by adamrc, but my Wan interface never connects on reboot. I can get the process to work, but it takes a couple steps...

I have to ...
login to the system console
kill all wpa_supplicant processes
delete /var/run/wpa_supplicant/igb0
run 04-wpa

refresh the interface in the GUI, after these steps it will load an external IP.
I'm beginning to wonder if there isn't a race condition causing the failure, but I thought the "04-" would keep that from happening.

The system hangs on Initializing WAN interface while booting.

Anyone have any ideas, or how to best troubleshoot?
Title: Re: [SOLVED]: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: Quasmo on September 13, 2024, 02:46:58 PM
For those seeing this. I was able to solve my problem. I had to make sure that the interface was enabled before I could run the WPA supplicant code.

I went to the 8311 discord and found the code posted here by Adamrc. There is a line omitted in the /usr/local/etc/rc.syshook.d/early/04-wpa file.


#!/bin/sh
/sbin/ifconfig igb0 link FF:FF:FF:FF:FF:FF

env OPENSSL_CONF=/conf/wpa/openssl.conf /usr/local/sbin/wpa_supplicant -Dwired -i igb0 -B -C /var/run/wpa_supplicant -c /conf/wpa/wpa_supplicant.conf


Where igb0 is the interface, and FF:FF:FF:FF:FF:FF is the Gateway MAC
Title: Re: [SOLVED]: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: hieroglyph on September 14, 2024, 06:15:09 PM
Quote from: adamrc on August 06, 2024, 09:18:08 PM
I'm back up and running as well.  I scrapped everything and went to the 8311 Discord channel for bypassing and got the proper info.  Here are the files and contents that I used.  Netgraph is no longer used/needed anymore.

/usr/local/etc/rc.syshook.d/early/04-wpa (make sure to chmod +x this file)

#!/bin/sh
env OPENSSL_CONF=/conf/wpa/openssl.conf /usr/local/sbin/wpa_supplicant -Dwired -i igb0 -B -C /var/run/wpa_supplicant -c /conf/wpa/wpa_supplicant.conf


/conf/wpa/openssl.conf

openssl_conf = openssl_init

[openssl_init]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Options = UnsafeLegacyRenegotiation
MinProtocol = TLSv1
CipherString = DEFAULT@SECLEVEL=0


/conf/wpa/wpa_supplicant.conf

# Generated by 802.1x Credential Extraction Tool
# Copyright (c) 2018-2019 devicelocksmith.com
# Version: 1.04 windows 386
#
# Change file names to absolute paths
ctrl_interface=DIR=/var/run/wpa_supplicant
openssl_ciphers=DEFAULT@SECLEVEL=0
eapol_version=2
ap_scan=0
fast_reauth=1
network={
        ca_cert="/conf/wpa/ca.pem"
        client_cert="/conf/wpa/client.pem"
        eap=TLS
        eapol_flags=0
        identity="REDACTED" # Internet (ONT) interface MAC address must match this value
        key_mgmt=IEEE8021X
        phase1="allow_canned_success=1"
        private_key="/conf/wpa/private.pem"
}


Quote from: Quasmo on September 13, 2024, 02:46:58 PM
For those seeing this. I was able to solve my problem. I had to make sure that the interface was enabled before I could run the WPA supplicant code.

I went to the 8311 discord and found the code posted here by Adamrc. There is a line omitted in the /usr/local/etc/rc.syshook.d/early/04-wpa file.


#!/bin/sh
/sbin/ifconfig igb0 link FF:FF:FF:FF:FF:FF

env OPENSSL_CONF=/conf/wpa/openssl.conf /usr/local/sbin/wpa_supplicant -Dwired -i igb0 -B -C /var/run/wpa_supplicant -c /conf/wpa/wpa_supplicant.conf


Where igb0 is the interface, and FF:FF:FF:FF:FF:FF is the Gateway MAC

These two posts got me back up and running.

The MAC addresses mentioned in the "04-wpa" and "wpa_supplicant.conf" files, is the MAC of the WAN port on the AT&T issued router. This same MAC address needs to be used in the opnsense GUI under Interfaces > Assignments > select your AT&T WAN interface > MAC address. On this same page look for Device at the top and take note of what it says. If it says something other than igb0, change the two places that say igb0 in 04-wpa to your network device name. Example: I had to change from igb0 to igc7 for my setup to work.
Title: Re: [SOLVED]: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: AryasDad on November 08, 2024, 10:36:48 PM
I cannot get this to work on my end.  I have followed the latest post additions and still am unable to bypass my BGW210.

I am suspecting my issue is I am running Opnsense as a VM in Proxmox and my setup for bridging my NIC to my VM is not setup properly?

Based on information I have seen online, my WAN interface on my Proxmox is setup as follows:

iface enp1s0 inet manual

iface enp1s0.0 inet manual

auto vmbr1
iface vmbr1 inet manual
bridge-ports enp1s0.0
bridge-stp off
bridge-fd 0
post-up echo 8 > /sys/class/net/vmbr1/bridge/group_fwd_mask


My Proxmox OPNsense VM WAN network is setup as follows:

net0: virtio=<BGW210 MAC address>,bridge=vmbr1,queues=8


And my various config files based on the above posts are as follows:
wpa_supplicant.conf

# Change file names to absolute paths
ctrl_interface=DIR=/var/run/wpa_supplicant
openssl_ciphers=DEFAULT@SECLEVEL=0
eapol_version=2
ap_scan=0
fast_reauth=1
network={
        ca_cert="/conf/wpa/CA.pem"
        client_cert="/conf/wpa/Client.pem"
        eap=TLS
        eapol_flags=0
        identity="<BGW210 MAC address>" # Internet (ONT) interface MAC address must match this value
        key_mgmt=IEEE8021X
        phase1="allow_canned_success=1"
        private_key="/conf/wpa/PrivateKey.pem"
}


openssl.conf

openssl_conf = openssl_init

[openssl_init]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Options = UnsafeLegacyRenegotiation
MinProtocol = TLSv1
CipherString = DEFAULT@SECLEVEL=0


/usr/local/etc/rc.syshook.d/early/04-wpa

#!/bin/sh

/sbin/ifconfig vtnet0 link <BGW210 MAC address>

env OPENSSL_CONF=/conf/wpa/openssl.conf /usr/local/sbin/wpa_supplicant -Dwired -i vtnet0 -B -C /var/run/wpa_supplicant -c /conf/wpa/wpa_supplicant.conf -t -dd


And finally, I set my WAN MAC to <BGW210 MAC address> in Opnsense GUI.

When I boot up, it hangs on WAN initialization and when I try to manually start 04-wpa, it fails.

Any idea what I need to adjust to get the working on my end?  I am running Opnsense version 24.7.7-amd64.

Thank you!
Title: Re: [SOLVED]: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: AryasDad on November 10, 2024, 12:38:29 AM
I seemed to have solved my own issue.  Apparently my first Nic port is defective.  Finally noticed with trying different things and plugging and unplugging that the ethernet data lights were not lighting up when I plugged into Nic #1.  Moved everything over to another Nic and BAM, it started negotiating and working as expected on a quick test.

So I have put everything back to previous still using the RG IP passthrough for now until I get a chance to properly adjust everything on a permanent basis.

Sometimes it is the simplest of things....  :-[
Title: Re: [SOLVED]: 24.7 Upgrade from 24.1.10 broke my AT&T Fiber Bypass
Post by: schlumpy on December 10, 2024, 07:06:29 AM
Quote from: Quasmo on September 11, 2024, 08:33:46 PM
Anyone have any ideas, or how to best troubleshoot?
Instead of ./early/04-wpa try moving it to ./start/94-wpa