Opnsense PPPoE config for Bell Fibe (FTTH) direct ONT connection with 2.5Gbps

[rekriux]

This is to share how I got to setup my current system for Bell fibe to bypass the HH3000 all-in-one router. It serves as a reminder and to share my story as I had to restart quite a few times... Overall, I spent a few weeks to get all in working order and to learn as I go. I still need to clean my network cable mess and relocate firewall and switch to a better place !

To get 2.5Gbps sync you need the Broadcom 57810S or a UBIQUITI ES-16-XG Edge Switch (~50$CAD or ~700$CAD) but you will be limited to ~1.2Gbps by Bell for your 1Gbps Fibe connection.

I have to say that I quite like the quality of the service and I think that most of the time it's the servers that can't send more than 200mbps to my box. Had a few long waits for Fedora downloads But Steam almost always top. Xbox service is as bad as it gets, hard to get over 100mbps.

References :
https://www.dslreports.com/forum/r31118482-Yes-you-CAN-bypass-the-HomeHub-3000
https://www.dslreports.com/forum/r32230041-Internet-Bypassing-the-HH3K-up-to-2-5Gbps-using-a-BCM57810S-NIC

My setup :
Dedicated Opnsense host :
    BIOSTAR A68N-5545 AMD A8-5545 (Quad core 1.7G, turbo 2.7G) [The included fan fits in a 1U!]
    200W 1U PSU (on hand; will upgrade to picopsu)
    iStarUSA D-118V2-ITX
    2x 2GB DDR3
    Dell Y40PH Broadcom 57810S Dual Port 10GbE SFP+ Converged Network Low Profile (ebay)
    Micron RealSSD 100Gb
    13000RPM 40x40 FAN !!! (on hand) It gets damn hot in a small 1u chassis even with a notebook APU of 19W. The network card and the ONT produces lots of heat also. The fan is on a split with the CPU fan so the MB controls both; case fan would always run 100% and is a jet engine of a few Watts.
    ---
   USB drive for booting install
   Keyboard + Screen (or iKVM) for initial config

   Cleaner Pen SC (in case you touch something with the fiber end. But I heard (ebay ones) that they are not real good and Flux brand is 10X price and 1000X better)

NOTICE
This is a "as best as I can" quick guide to setting up a connection to Bell.ca while removing the modem HH3K. You will have to provide WIFI with an other modem (openwrt on WNDR3800 for me) and enable IPTV (will be added later or see pfsense exemples online).

NOTE: I had poor speedtest-cli results when run on the opnsense router. Performances are way better on my workstation. I lost quite some time on it. The APU is able to push gigabit speed on PPPoE but not while running the speedtest-cli client.

INSTALLATION :

Setting up the router host

#Start by downloading the opnsense vga image and writing it to USB (from linux host)
bunzip2 OPNsense-19.7-OpenSSL-vga-amd64.img.bz2
sudo dd if=OPNsense-19.7-OpenSSL-vga-amd64.img of=/dev/sdX bs=1M

# For now, keep the bell modem working and hook re0 from opnsense to it
# This is for setting up all the things first before actually trying anything

# Boot your next to be opnsense box
# Press ctrl-s when prompt by 10G card
# Disable boot prompt and dhcp for iscsi for both connector
# Reboot and boot USB (F9 to select boot device with Biostar)

# Login “installer” with password “opnsense”
# Install with GPT and defaults options
# Reboot

# Login
# Assign interface WAN->re0 ; get dhcp address
# DON'T Assign interface LAN or you will not be able to access web
# Note your IP

# Open your browser from your computer to https://IP
# Skip wizard
# Enable ssh login System->Settings->Administration

# Enable WAN access to webgui so that when you add a second interface, you don't get locked out
# Firewall->Rules->WAN


# Go Interfaces->Other Types->VLAN
# Create a new VLAN 35
# Add 36 for TV (37 VOIP?)


# Go  Interfaces->Assignments and create WANBell from bxe0_vlan35
# Note that you MUST have made the firewall rule before !
# Do not create a Point-to-Point connection manually, it didn't work and I was going nowhere and had to restart from start.
# By selecting PPPoE as IPv4 configuration, it will create the needed Point-to-Point connection.
# Else you will be like https://forum.opnsense.org/index.php?topic=8633.0


# Add a new interface for port2 of 10Gbe card to network and configure dhcpd on it

# Save and apply changes
# Update system to latest
# Reboot to make sure everything is alright. Poweroff

# Connect ONT to port0 of 10Gbe card
# Power on
# Check that all works !



PATCHING the bxe driver to enable 2.5G

Ref: https://www.dslreports.com/forum/r32230041-Internet-Bypassing-the-HH3K-up-to-2-5Gbps-using-a-BCM57810S-NIC~start=60

# Do the ediag fix mentioned in the post above. I had to do a efi usb disk as I couldn't get it done with freedos.
# Do it only for ONE port. All of this is not needed if you have spent 700$CAD for the switch
# I am looking at other 10Gbe switch, Mikrotik doesn't support 2.5Gbps but is a good candidate.

On a FreeBSD development PC running SAME KERNEL (ran in a VM)

pkg install git
cd /usr
git clone https://github.com/opnsense/tools
cd tools
make update

cd /usr/src/sys/dev/bxe
# Get the updated files from FreeBSD as the patch was merged. Don't use later version as it will have unmet dependencies.
curl https://raw.githubusercontent.com/freebsd/freebsd/6c5a0d70113cc6857a91fda663d34e4cd7744ac5/sys/dev/bxe/bxe_stats.c > bxe_stats.c
curl https://raw.githubusercontent.com/freebsd/freebsd/6c5a0d70113cc6857a91fda663d34e4cd7744ac5/sys/dev/bxe/bxe.h > bxe.h
curl https://raw.githubusercontent.com/freebsd/freebsd/6c5a0d70113cc6857a91fda663d34e4cd7744ac5/sys/dev/bxe/bxe.c > bxe.c

cd /usr/src/sys/modules/bxe
make

On opnsense router

# Log to opnsense and make a backup of the driver
cp /boot/kernel/if_bxe.ko ~/if_bxe.ko.ori
# Force load module
echo 'if_bxe_load="YES"' >> /boot/loader.conf.local
# Fix states issues
echo 'hw.bxe.interrupt_mode="1"' >> /boot/loader.conf.local
echo 'net.inet.tcp.tso="0"' >> /boot/loader.conf.local
echo 'if_bxe_load="YES"' >> /boot/loader.conf.local

# Make sure Opnsense doesn't overwrite those files; turn to noschg if you want to change them
chflags schg /boot/kernel/if_bxe.ko
chflags schg /boot/loader.conf.local

# Copy file from VM to opnsense
scp root@VM:/usr/src/sys/modules/bxe/if_bxe.ko /boot/kernel

#Check dmesg for linkspeed of 2500 
# bxe0: NIC Link is Up, 2500 Mbps full duplex, Flow control: ON - receive & transmit

# Test speed from a connected computer (10Gbe if possible)

# test speed
speedtest-cli --server 17567
https://fast.com
https://www.speedtest.net
My speed test was done on cat5e connection as I didn't upgrade my home network to 10Gb (yet! soon coming:)

IPTV
Reference :
https://forum.netgate.com/topic/78892/how-to-get-bell-fibe-in-quebec-ontario-internet-and-iptv-working-with-pfsense/80

I went with a BRIDGE between my 2 SFP+ ports of the network card and created 2 vlans (wan36 and lan36). I made a few different things from the pfsense tutorial, but it works. IPTV packets from Bell to IPTV receiver must be taged vlan36. I also have a switch in between the two. You could also make it work over an cat5e ethernet cable instead of the SFP+. So here we go :

Create vlans :

In Interfaces->Other Types->VLAN

Assign vlans to interfaces :


Assign DHCP to wan_iptv


Create Bridge between vlans to enable routing.


Install and configure service IGMP Proxy


Configure firewall :



Note that IPTV DOESN'T needs IPv6 to work.


That should be it !
I had to use Firewall->Log View->Plain View to look at packets being blocked/allowed from interfaces to debug. If you have any problems, check there.

Questions

You may ask questions below. I will sum them up if answers are provided by someone else, but I might not be able to figure things for you. I hope I didn't skip any part, so if you implement it on your side, please leave comments on any obscure parts.
Be SAFE !

[fegauthier]

Hi rekriux,

First of all, thanks for you guide!

Which GPON SFP that Bell gives you? Does it work with the Nokia G-010S-A?

Thanks

[Sisyphe]

Hi there,

Great guide! Do you know if the patched bxe driver is now included in OpnSense 20.7 or does it still need to be modified?

Cheers