Lessons Learned deploying OPNsense 2022-10-01

[lordjakoby]

Successful venture #1 || 2.5Gbps WAN Upgrade, 2Gbps  (x2 1Gbps) LACP to LANs


Summary
Documenting a successful venture in running a 2.5Gbps WAN, upgraded from 1Gbps. Coupled with x2 1Gbps WAN LACP to a Cisco 1000 series  (C1000-8P-E-2G-L).

Primary motivator was to take advantage of 1.2gbps Xfinity uplink.
The card I installed: https://www.amazon.com/NICGIGA-Network-Adapter-RTL8125B-Ethernet/dp/B09HGRK5XB/ref=sr_1_13?keywords=2.5gbps%2Bethernet%2Badapter&qid=1664658077&qu=eyJxc2MiOiIyLjgxIiwicXNhIjoiMS44NiIsInFzcCI6IjEuMDAifQ%3D%3D&sprefix=2.5Gbps%2Bether%2Caps%2C139&sr=8-13&th=1

The interface was not recognized upon booting, but performed the following bash history to fix. For a while now OPNsense has the realtek drivers included but just need to be included.

##########################################################
     1  18:23   uname -rs
     2  18:35   ls
     3  18:35   cd /
     4  18:35   ls
     5  18:36   ls
     6  18:36   cd ~
     7  18:36   ls
     8  18:36   pwd
     9  18:36   cd ..
    10  18:36   mv realtekdrivers.zip /root
    11  18:37   cd ~
    12  18:37   ls
    13  18:37   unzip realtekdrivers.zip
    14  18:37   ls -alh
    15  18:37   ld
    16  18:37   ls
    17  18:37   ls -alh realtekdrivers/1
    18  18:37   ls -alh realtekdrivers/
    19  18:38   cp realtekdrivers/12.2/1.96.04/if_re.ko /boot/kernel
    20  18:39   chown root:wheel /boot/kernel/if_re.ko
    21  18:39   chmod 0555 /boot/kernel/if_re.ko
    22  18:41   nano /boot/loader.conf.local
    23  18:41   vi /boot/loader.conf.local
    24  18:43   cat /boot/loader.conf.local
    25  18:44   pkg install nano
    26  18:44   kldload if_re.ko
    27  18:48   pkg install nano
    28  21:28   history
    29  21:31   pkg install os-realtek-re
    30  21:35   nano /boot/loader.conf
    31  21:36   nano /boot/loader.conf.local
    32  21:38   ls -la /boot/kernel/if_re.ko
    33  21:38   ls -la /boot/kernel/
    34  21:38   ls -la /boot/kernel/ | less
    35  21:39   ls -la /boot/kernel/ | less
    36  21:39   kldload if_re.ko
    37  21:39   exit
    38  17:06   ifconfig -m -v lagg0
    39  17:06   exit
    40  14:29   history
##########################################################


Successful venture #2 || Site to Site VPN with ZeroTier


Installed another OPNsense box at another site I run and needed direct L3 reachability. Installed zerotier packages on both OPNsense firewalls > added both to the same network ID > added interfaces, ZT assigned IP addresses and allow rules for each > added routes in the ZeroTier cloud controller. (add destination networks for each site/each sites LANs, and point the next hop as the appropriate zerotier interface.)

At this point I encountered a strange issue where the zerotier interface showed extremly high usage (~3TB) over a couple minutes. After significant troubleshooting, appears the issue was that a loop occurs when using non WAN paths to re-create zerotier tunnel.
Trick was to add a blacklist preventing for zerotier tunnels forming through the FW's local LAN's.

Login to OPNsense GUI > VPN > ZeroTier > Settings > local.conf settings | paste in:
{
   "physical": {
      "10.1.212.0/24": { "blacklist": true },
                "10.17.17.0/24": {"blacklist":true }
   }
}

This will prevent the looping and stabilize the tunnel.






OPNsense Hardware

Motherboard: Supermicro X8STi
CPU: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz (6 cores, 12 threads)
RAM: 24GB
SSD: 120GB Kingston

[sunmast]

Hi, did you enable the hardware acceleration? Is it working? Thanks.