Messages

[Summary]

Successful venture #1 || 2.5Gbps WAN Upgrade, 2Gbps  (x2 1Gbps) LACP to LANs


Summary
Documenting a successful venture in running a 2.5Gbps WAN, upgraded from 1Gbps. Coupled with x2 1Gbps WAN LACP to a Cisco 1000 series  (C1000-8P-E-2G-L).

Primary motivator was to take advantage of 1.2gbps Xfinity uplink.
The card I installed: https://www.amazon.com/NICGIGA-Network-Adapter-RTL8125B-Ethernet/dp/B09HGRK5XB/ref=sr_1_13?keywords=2.5gbps%2Bethernet%2Badapter&qid=1664658077&qu=eyJxc2MiOiIyLjgxIiwicXNhIjoiMS44NiIsInFzcCI6IjEuMDAifQ%3D%3D&sprefix=2.5Gbps%2Bether%2Caps%2C139&sr=8-13&th=1

The interface was not recognized upon booting, but performed the following bash history to fix. For a while now OPNsense has the realtek drivers included but just need to be included.

##########################################################
     1  18:23   uname -rs
     2  18:35   ls
     3  18:35   cd /
     4  18:35   ls
     5  18:36   ls
     6  18:36   cd ~
     7  18:36   ls
     8  18:36   pwd
     9  18:36   cd ..
    10  18:36   mv realtekdrivers.zip /root
    11  18:37   cd ~
    12  18:37   ls
    13  18:37   unzip realtekdrivers.zip
    14  18:37   ls -alh
    15  18:37   ld
    16  18:37   ls
    17  18:37   ls -alh realtekdrivers/1
    18  18:37   ls -alh realtekdrivers/
    19  18:38   cp realtekdrivers/12.2/1.96.04/if_re.ko /boot/kernel
    20  18:39   chown root:wheel /boot/kernel/if_re.ko
    21  18:39   chmod 0555 /boot/kernel/if_re.ko
    22  18:41   nano /boot/loader.conf.local
    23  18:41   vi /boot/loader.conf.local
    24  18:43   cat /boot/loader.conf.local
    25  18:44   pkg install nano
    26  18:44   kldload if_re.ko
    27  18:48   pkg install nano
    28  21:28   history
    29  21:31   pkg install os-realtek-re
    30  21:35   nano /boot/loader.conf
    31  21:36   nano /boot/loader.conf.local
    32  21:38   ls -la /boot/kernel/if_re.ko
    33  21:38   ls -la /boot/kernel/
    34  21:38   ls -la /boot/kernel/ | less
    35  21:39   ls -la /boot/kernel/ | less
    36  21:39   kldload if_re.ko
    37  21:39   exit
    38  17:06   ifconfig -m -v lagg0
    39  17:06   exit
    40  14:29   history
##########################################################


Successful venture #2 || Site to Site VPN with ZeroTier


Installed another OPNsense box at another site I run and needed direct L3 reachability. Installed zerotier packages on both OPNsense firewalls > added both to the same network ID > added interfaces, ZT assigned IP addresses and allow rules for each > added routes in the ZeroTier cloud controller. (add destination networks for each site/each sites LANs, and point the next hop as the appropriate zerotier interface.)

At this point I encountered a strange issue where the zerotier interface showed extremly high usage (~3TB) over a couple minutes. After significant troubleshooting, appears the issue was that a loop occurs when using non WAN paths to re-create zerotier tunnel.
Trick was to add a blacklist preventing for zerotier tunnels forming through the FW's local LAN's.

Login to OPNsense GUI > VPN > ZeroTier > Settings > local.conf settings | paste in:
{
   "physical": {
      "10.1.212.0/24": { "blacklist": true },
                "10.17.17.0/24": {"blacklist":true }
   }
}

This will prevent the looping and stabilize the tunnel.






OPNsense Hardware

Motherboard: Supermicro X8STi
CPU: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz (6 cores, 12 threads)
RAM: 24GB
SSD: 120GB Kingston

[ifconfig igb0 promisc; ifconfig igb0_vlan2 promisc; ifconfig igb0_vlan3 promisc; ifconfig igb0_vlan4 promisc; ifconfig igb0_vlan5 promisc; ifconfig igb0_vlan6 promisc;]

Want to say thank you both for this discussion. I spent this weekend troubleshooting my home network eventually realizing that the lan sub interfaces shared the physical NICs/parent mac address once I saw the same mac address for each gateway in each vlan's mac-address table. This was causing strange issues like allowing me to initially ssh into a machine and then disconnecting me 5-8 seconds later with cli saying "Operation timed out.", and fw logs saying blocked ssh via default deny rule after it had already correctly hit the allow rules setup for the vlan interface. The lan networks were setup for allow any/any for troubleshooting and was perplexed at how the rules could work, then my ssh traffic no longer match the allow and start hitting the default deny.

Wanted to consolidate the steps mentioned above as I had a hard time finding anything useful till this lovely post.

Steps that fixed my router on a stick via a single physical port opnsense implementation. Downlink is a cisco trunk port

- initial setup found here: https://www.youtube.com/watch?v=dv13d6rfQPI
- enabled ssh access via: https://techexpert.tips/opnsense/opnsense-remote-access-ssh/
(i had disabled ssh initially, but wasn't easily able to get my admin non-root user to work so temporarily setup root ssh.)

- ran: opnsense-patch 55f54f8
output:
root@eri-hq-opnsense:~ # opnsense-patch 55f54f8
Fetched 55f54f8 via https://github.com/opnsense/core
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 55f54f878e88fcd67d462f7b78b151067f48296a Mon Sep 17 00:00:00 2001
|From: Maurice Walker <maurice@walker.earth>
|Date: Wed, 3 Mar 2021 22:22:37 +0100
|Subject: [PATCH] Fine-grained MAC address spoofing for VLANs
|
|- If a spoofed MAC address is configured for a VLAN, apply it to the VLAN only, not the parent.
|- If a spoofed MAC address is configured for a parent interface, restore any spoofed VLAN MAC addresses.
|---
| src/etc/inc/interfaces.inc | 13 ++++++++-----
| 1 file changed, 8 insertions(+), 5 deletions(-)
|
|diff --git a/src/etc/inc/interfaces.inc b/src/etc/inc/interfaces.inc
|index b7567e958e..6f6d08ba4f 100644
|--- a/src/etc/inc/interfaces.inc
|+++ b/src/etc/inc/interfaces.inc
--------------------------
Patching file etc/inc/interfaces.inc using Plan A...
Hunk #1 succeeded at 2399 (offset -2 lines).
done
All patches have been applied successfully.  Have a nice day.
#####################################

- ran these commands in cli, will remain until reboot.

ifconfig igb0 promisc; ifconfig igb0_vlan2 promisc; ifconfig igb0_vlan3 promisc; ifconfig igb0_vlan4 promisc; ifconfig igb0_vlan5 promisc; ifconfig igb0_vlan6 promisc;


For a reboot persistent config, I originally added the commands to /etc/rc.local but for w/e reason it wasn't working for me after reboot.

What did work however was to add the commands to the rc file, located at /usr/local/etc/rc
between the last "exit 0" line and after "/usr/local/etc/rc.subr.d/livemode" see below live example.

I found it was personally easier to scp the file to my machine (vi default in opnsense) > edit with nano > delete the opnsense /usr/local/etc/rc  and re-upload the rc file.

#copy from opnsense
scp root@10.11.254.10:/usr/local/etc/rc rc

#copy back to opnsense
scp rc root@10.11.254.10:/usr/local/etc/rc

-----------------------> Example of: /usr/local/etc/rc <---------------------------

/usr/local/etc/rc.subr.d/livemode

#CUSTOM

#Router on a stick requires promiscous mode enable on all participating #interfaces. Including parent.
ifconfig igb0 promisc; ifconfig igb0_vlan2 promisc; ifconfig igb0_vlan3 promisc; ifconfig igb0_vlan4 promisc; ifconfig igb0_vlan5 promisc; ifconfig igb0_vlan6 promisc;

exit 0
root@eri-hq-opnsense:~ # cat /usr/local/etc/rc

----------------------------------> Example end <----------------------------------

- changed sub vlan interfaces to unique mac addresses in the gui.

- confirmed each gateways new mac address was only showing in the associated vlans mac address table.

#########################################
Previous:
eri-hq-sw1#sh mac add dynamic int g1/0/1
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
   2    4062.3114.0cf4    DYNAMIC     Gi1/0/1
   3    4062.3114.0cf4    DYNAMIC     Gi1/0/1
   4    4062.3114.0cf4    DYNAMIC     Gi1/0/1
   5    4062.3114.0cf4    DYNAMIC     Gi1/0/1
   6    4062.3114.0cf4    DYNAMIC     Gi1/0/1
   7    4062.3114.0cf4    DYNAMIC     Gi1/0/1

After fix:
eri-hq-sw1#sh mac add int g1/0/1
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
   2    4062.3114.0cc2    DYNAMIC     Gi1/0/1
   3    4062.3114.0cc3    DYNAMIC     Gi1/0/1
   4    4062.3114.0cc4    DYNAMIC     Gi1/0/1
   5    4062.3114.0cc5    DYNAMIC     Gi1/0/1
   6    4062.3114.0cc6    DYNAMIC     Gi1/0/1
   7    4062.3114.0cf4    DYNAMIC     Gi1/0/1