Bypass AT&T Residential Gateway using Supplicant Mode

[wleeb1010]

I am an AT&T fiber customer and wanted to bypass my residential gateway (RG) and connect the OPNsense WAN interface directly to my ONT. I am running OPNsense 20.1 as a VM on an ESXi 7.0.1 host. I have read the many ways of bypassing including bridge and supplicant modes. I decided to go with supplicant mode as it is less complicated, with fewer components required, AND it gets rid of my RG completely (saving me 36 watts running 24/7). Supplicant mode does not use complicated Netgraph scripting that bridge mode requires. All supplicant mode needs is to call a single line of code (/usr/sbin/wpa_supplicant -s -B -Dwired -ivmx2 -c/etc/wpa_supplicant.conf &).

I am not the one that figured this out, I am merely aggregating multiple people's efforts for other OPNsense users that may want to try this while leaving myself a bread trail in case I need to repeat this in the future.

For supplicant mode, you will need the 802.1x certificates located on the RG as well as the MAC address of the ONT port. The certificates are stored as .der binary files on the RG and will need to be converted to .pem files. Accessing the filesystem will require downgrading the RG firmware so PROCEED AT YOUR OWN RISK.

I pulled the 802.1x certificates and ONT MAC address from my BGW210-700 gateway using the instructions outlined here: https://www.reddit.com/r/ATT/comments/g59rwm/bgw210700_root_exploitbypass/. When I first tried this, I pulled the certs from my RG and waited until the next day to finish. The certs failed to authenticate as AT&T had updated the certs on my RG/ONT overnight. Since the process defined above to pull the certs leaves the RG with permanent telnet access via TCP port 28, I logged back in and grabbed the new certs and pushed them to OPNsense which resolved the issue.

To convert the .der binaries to .pem files and obtain the ONT MAC, I downloaded mfg_dat_decode.exe found here https://github.com/iwleonards/extract-mfg. Place the *.der files and mfg.dat, you downloaded from the RG in the previous step, and the mfg_dat_decode.exe file in the same folder. Open a command prompt from that folder and execute the mfg_dat_decode.exe. This will convert the *.der files to *.pem files and create the wpa_supplicant.conf file. Edit the wpa_supplicant.conf file making note of the ONT MAC address then change the file paths for the certificates and private keys to their absolute path e.g. /etc/CA_xxxxxx-xxxxxxxxxx.pem, /etc/Client_xxxxxx-xxxxxxxxxx.pem and /etc/private_key_xxxxxx-xxxxxxxxxx.pem. Copy the wpa_supplicant.conf and the 3 *.pem files to the /etc/ folder in OPNsense.

Create a new vSwitch in ESXi and connect it to an unused NIC on the host. Set the vSwitch Security settings to accept Promiscuous mode, MAC address changes and Forged transmits. Create a new port group and set the VLAN ID to 0 and inherit the Security settings from the vSwitch. Add a new network adapter (VMXNET 3) to OPNsense VM and manually assign it the MAC address from the ONT port of the RG and connect it to the newly created port group. Note that the VM must be shutdown to manually assign MAC to NIC adapter. Connect your ONT to the newly configured ESXi host port with a patch cable.

Power on the OPNsense VM and SSH into the OPNsense console. Run ifconfig to verify the new interface is shown with the properly assigned ONT MAC address. If the interface is not listed, it is likely due to an ESXi bug associated with adding multiple VMXNET adapters to a FreeBSD 11 VM that improperly assigns the ethernetx.pciSlotNumber as -1. See ESXi ISSUE below for resolution.

Create a new script named 99-opnatt in the /usr/local/etc/rc.syshook.d/early/ directory of the OPNsense VM.
Place this code in the file. Set the ONT_IF variable to the interface you are connecting to the ONT, mine was vmx2:

#!/bin/sh
set -e

ONT_IF='vmx2'
LOG=/var/log/opnatt.log

getTimestamp(){
    echo `date "+%Y-%m-%d %H:%M:%S :: [opnatt] ::"`
}

{
    echo "$(getTimestamp) Bypass AT&T Residential Gateway using supplicant mode"
    echo "$(getTimestamp) Command: /usr/sbin/wpa_supplicant -s -B -Dwired -i$ONT_IF -c/etc/wpa_supplicant.conf &"
    echo -n "$(getTimestamp) Executing command... "
    /usr/sbin/wpa_supplicant -s -B -Dwired -i$ONT_IF -c/etc/wpa_supplicant.conf &
    echo ""
} >> $LOG

Make the 99-opnatt file executable with the following command:
chmod +x /usr/local/etc/rc.syshook.d/early/99-opnatt
This script will automatically be executed on startup of OPNsense prior to network being initialized.

From the OPNsense GUI, assign the new interface to the WAN port. Also, set the MAC address of the WAN interface to the RG ONT MAC address. Reboot OPNsense and verify the AT&T Public IP has been assigned to the WAN interface via DHCP. Note that the DHCP lease is for 1 hour and after that it fails to renegotiate a new lease so the WAN interface looses the public IP. While troubleshooting the issue, I noticed that if I reset (disable/enable) the OPNsense WAN interface from the GUI, it would successfully negotiate a new DHCP lease and all subsequent leases from that point on. I wanted to automate this so see WAN RESET WORKAROUND below.

ESXi ISSUE:
I had to shutdown the VM and edit the vmname.vmx file for the VM on the ESXi host datastore. First I deleted the vmname.vmx.lck and vmname.vmx~ files. I then edited the vmname.vmx file changing ethernet0.pciSlotNumber to 1184, ethernet1.pciSlotNumber to 2208 and ethernet2.pciSlotNumber to 192. My configuration has the SCSI0 interface pciSlotNumber assigned 160, so I started with 1184 for eternet0. Depending on your VM configuration, you may have to use different numbers. Checkout the post from Shawn on this thread for proper pciSlotNumbers: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198406. Once the pciSlotNumbers have been changed, power on OPNsense VM then log into the OPNsense console via SSH. Run ifconfig to verify the new interface is shown with the correctly assigned ONT MAC address.

WAN RESET WORKAROUND:
To automate the reset of the WAN interface, I used the Monit service from the OPNsense GUI. Monit can be setup to ping an IP address and, on ping failure, execute a script. I used a builtin script (/usr/local/etc/rc.configure_interface) to handle the interface reset.

Here are the steps to configure Monit:
Under Services>>Monit>>Settings>>Service Tests Settings tab add a new service test
Name the service test: WAN_CHECK
Put this code in the Condition field: failed ping4 count 1 address 192.168.11.1 Replace IP with OPNsense LAN IP
For Action choose: Execute
For Path put: /usr/local/etc/rc.configure_interface wan
Now add a new service from the Services>>Monit>>Settings>>Service Settings tab
Name the service: WAN_RESTART
For Type choose: Remote Host
For address put: 1.1.1.1. or whatever external public IP you want to monitor.
For Tests select: WAN_CHECK
Now enable Monit from the Services>>Monit>>Settings>>General Settings tab

TODO
I do not understand why, after the initial reboot, 1 hour later DHCP fails to negotiate a new lease and simply resetting the WAN interface once resolves it permanently.  Post a reply if you know why this is the case as I would love to not have to use the WAN reset workaround.

[GPz1100]

Did you ever get the dhcp issue resolved?