Successful venture #1 || 2.5Gbps WAN Upgrade, 2Gbps (x2 1Gbps) LACP to LANs
Summary
Documenting a successful venture in running a 2.5Gbps WAN, upgraded from 1Gbps. Coupled with x2 1Gbps WAN LACP to a Cisco 1000 series (C1000-8P-E-2G-L).
Primary motivator was to take advantage of 1.2gbps Xfinity uplink.
The card I installed: https://www.amazon.com/NICGIGA-Network-Adapter-RTL8125B-Ethernet/dp/B09HGRK5XB/ref=sr_1_13?keywords=2.5gbps%2Bethernet%2Badapter&qid=1664658077&qu=eyJxc2MiOiIyLjgxIiwicXNhIjoiMS44NiIsInFzcCI6IjEuMDAifQ%3D%3D&sprefix=2.5Gbps%2Bether%2Caps%2C139&sr=8-13&th=1
The interface was not recognized upon booting, but performed the following bash history to fix. For a while now OPNsense has the realtek drivers included but just need to be included.
##########################################################
1 18:23 uname -rs
2 18:35 ls
3 18:35 cd /
4 18:35 ls
5 18:36 ls
6 18:36 cd ~
7 18:36 ls
8 18:36 pwd
9 18:36 cd ..
10 18:36 mv realtekdrivers.zip /root
11 18:37 cd ~
12 18:37 ls
13 18:37 unzip realtekdrivers.zip
14 18:37 ls -alh
15 18:37 ld
16 18:37 ls
17 18:37 ls -alh realtekdrivers/1
18 18:37 ls -alh realtekdrivers/
19 18:38 cp realtekdrivers/12.2/1.96.04/if_re.ko /boot/kernel
20 18:39 chown root:wheel /boot/kernel/if_re.ko
21 18:39 chmod 0555 /boot/kernel/if_re.ko
22 18:41 nano /boot/loader.conf.local
23 18:41 vi /boot/loader.conf.local
24 18:43 cat /boot/loader.conf.local
25 18:44 pkg install nano
26 18:44 kldload if_re.ko
27 18:48 pkg install nano
28 21:28 history
29 21:31 pkg install os-realtek-re
30 21:35 nano /boot/loader.conf
31 21:36 nano /boot/loader.conf.local
32 21:38 ls -la /boot/kernel/if_re.ko
33 21:38 ls -la /boot/kernel/
34 21:38 ls -la /boot/kernel/ | less
35 21:39 ls -la /boot/kernel/ | less
36 21:39 kldload if_re.ko
37 21:39 exit
38 17:06 ifconfig -m -v lagg0
39 17:06 exit
40 14:29 history
##########################################################
Successful venture #2 || Site to Site VPN with ZeroTier
Installed another OPNsense box at another site I run and needed direct L3 reachability. Installed zerotier packages on both OPNsense firewalls > added both to the same network ID > added interfaces, ZT assigned IP addresses and allow rules for each > added routes in the ZeroTier cloud controller. (add destination networks for each site/each sites LANs, and point the next hop as the appropriate zerotier interface.)
At this point I encountered a strange issue where the zerotier interface showed extremly high usage (~3TB) over a couple minutes. After significant troubleshooting, appears the issue was that a loop occurs when using non WAN paths to re-create zerotier tunnel.
Trick was to add a blacklist preventing for zerotier tunnels forming through the FW's local LAN's.
Login to OPNsense GUI > VPN > ZeroTier > Settings > local.conf settings | paste in:
{
"physical": {
"10.1.212.0/24": { "blacklist": true },
"10.17.17.0/24": {"blacklist":true }
}
}
This will prevent the looping and stabilize the tunnel.
OPNsense Hardware
Motherboard: Supermicro X8STi
CPU: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz (6 cores, 12 threads)
RAM: 24GB
SSD: 120GB Kingston
Hi, did you enable the hardware acceleration? Is it working? Thanks.