[NOT OPNsense issue] Routing table breaks when I add a new 10Gig Interface

[WholesomeTRex]

Hi all, I tried searching but didn't find what I'm looking for.

Recently I bought a new Brocade ICX6610-48P switch so that I can start using 10gig connections between OPNsense, my NAS, and the rest of my network. I now have a SFP+ 10G card in my Proxmox host which runs OPNsense. I created the vmbridge in Proxmox and added that to OPNsense. That's all good.

I run into an issue where when I add and enable the new 10Gig interface inside of OPNsense, pretty much immediately it breaks routing. Even after restarting OPNsense, routing is broken... to the point where I can ping some devices on a VLAN, but not others on the same VLAN. (mind you, no firewall rules have changed with this addition... just adding the interface). Once I've removed the new 10G interface from OPNsense, I've got to restart my current TP-Link core switch and OPNsense for routing to not get stuck.

Currently, I am planning to just use the Brocade switch as a higher speed Layer 2 switch for the time being and not perform L3 routing on it.... leaving that to OPNsense for now. So I've been configuring all the VLAN's on the Brocade switch so it'll be able to pass traffic between devices on the same VLAN instead of hitting OPNsense to route heavy storage traffic. Eventually, I will be completely removing my existing TP-Link SG2428P switch that I'm using as my core switch and all the currently connected devices will be moved over to the Brocade switch. I realize the currently, it's effectively 2 separate LAN's since the switches are not connected (so that I can avoid causing a network loop from having the 2 switches connected together AND both connected to OPNsense.

What I'm looking for is guidance on how to proceed and not mess up my network.

• How do I add this 10Gig interface to OPNsense and then start moving VLAN's over to it AND not break routing?
• Once I have this new 10G interface setup and working, should I just create "new" VLAN's on this new interface with the same VLAN ID's so that I don't break all the routing between my current TP-Link switch and my new Brocade switch?

   

[EricPerl]

Adding/enabling the new interface shouldn't mess the rest of the network if you use a disjoint subnet.
You could also not assign the physical device. Then you don't have untagged traffic on that network. This is the recommended configuration.

The method used to move over the VLANs depends on your overall topology.
Moving an entire VLAN could be near impossible if you had a second layer of switches for example, or even a VLAN aware AP.

If you only have access ports on the switches, then you can reassign a VLAN interface from one vlan device on vtnetX to another vlan device on vtnetY.
All the configuration tied to the interface will follow.
Then you move all cables for that VLAN accordingly.

It seems putting a third switch in front (full trunks) would make things easier. Configure access port on the "new" side, move cable. And so on.
In theory, I guess you could bridge the old NIC and new NIC during the transition, but it may not be worth the hassle (build then tear down).
Bridges with VLANs can be tricky given the number of threads on the subject.

[WholesomeTRex]

I've made some progress in figuring out what the issue is. The card is up, the connection to the switch is recognized, but I believe that the Intel NIC does not like the DAC that I've used. As such, I continuously see the following messages in dmesg:

[ 1796.988152] ixgbe 0000:06:00.0 enp6s0: detected SFP+: 3
[ 1797.112179] vmbr1: port 1(enp6s0) entered disabled state
[ 1797.127164] ixgbe 0000:06:00.0 enp6s0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 1797.127191] vmbr1: port 1(enp6s0) entered blocking state
[ 1797.127194] vmbr1: port 1(enp6s0) entered forwarding state
[ 1797.540948] ixgbe 0000:06:00.0 enp6s0: Received ECC Err, initiating reset
[ 1797.540957] ixgbe 0000:06:00.0 enp6s0: Reset adapter

EDIT: After digging into more of this situation, this is not an OPNsense issue, and is a Proxmox issue with the Intel X520-10G 82599EN (and ES) SFP+ cards. Proxmox does not like the card and the Internet appears to be unsure whether this is an issue with NIC / DAC compatibility or if Proxmox itself does not have proper drivers for it. As such, no further effort is needed in attempting to resolve the issue in OPNsense forums.