Messages

I've made some progress in figuring out what the issue is. The card is up, the connection to the switch is recognized, but I believe that the Intel NIC does not like the DAC that I've used. As such, I continuously see the following messages in dmesg:

[ 1796.988152] ixgbe 0000:06:00.0 enp6s0: detected SFP+: 3
[ 1797.112179] vmbr1: port 1(enp6s0) entered disabled state
[ 1797.127164] ixgbe 0000:06:00.0 enp6s0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 1797.127191] vmbr1: port 1(enp6s0) entered blocking state
[ 1797.127194] vmbr1: port 1(enp6s0) entered forwarding state
[ 1797.540948] ixgbe 0000:06:00.0 enp6s0: Received ECC Err, initiating reset
[ 1797.540957] ixgbe 0000:06:00.0 enp6s0: Reset adapter

EDIT: After digging into more of this situation, this is not an OPNsense issue, and is a Proxmox issue with the Intel X520-10G 82599EN (and ES) SFP+ cards. Proxmox does not like the card and the Internet appears to be unsure whether this is an issue with NIC / DAC compatibility or if Proxmox itself does not have proper drivers for it. As such, no further effort is needed in attempting to resolve the issue in OPNsense forums.

Hi all, I tried searching but didn't find what I'm looking for.

Recently I bought a new Brocade ICX6610-48P switch so that I can start using 10gig connections between OPNsense, my NAS, and the rest of my network. I now have a SFP+ 10G card in my Proxmox host which runs OPNsense. I created the vmbridge in Proxmox and added that to OPNsense. That's all good.

I run into an issue where when I add and enable the new 10Gig interface inside of OPNsense, pretty much immediately it breaks routing. Even after restarting OPNsense, routing is broken... to the point where I can ping some devices on a VLAN, but not others on the same VLAN. (mind you, no firewall rules have changed with this addition... just adding the interface). Once I've removed the new 10G interface from OPNsense, I've got to restart my current TP-Link core switch and OPNsense for routing to not get stuck.

Currently, I am planning to just use the Brocade switch as a higher speed Layer 2 switch for the time being and not perform L3 routing on it.... leaving that to OPNsense for now. So I've been configuring all the VLAN's on the Brocade switch so it'll be able to pass traffic between devices on the same VLAN instead of hitting OPNsense to route heavy storage traffic. Eventually, I will be completely removing my existing TP-Link SG2428P switch that I'm using as my core switch and all the currently connected devices will be moved over to the Brocade switch. I realize the currently, it's effectively 2 separate LAN's since the switches are not connected (so that I can avoid causing a network loop from having the 2 switches connected together AND both connected to OPNsense.

What I'm looking for is guidance on how to proceed and not mess up my network.

• How do I add this 10Gig interface to OPNsense and then start moving VLAN's over to it AND not break routing?
• Once I have this new 10G interface setup and working, should I just create "new" VLAN's on this new interface with the same VLAN ID's so that I don't break all the routing between my current TP-Link switch and my new Brocade switch?

   

So, what's the exact error msg when
A) ping public site  - ping - "request timed out" when pinging google.com
B) opening a website - Internal websites: the connection has timed out. External websites: "Hmm. we're having trouble finding that site"

for external websites, sounds  like it can't query DNS, but nslookup google.com returns valid results from my AdGuard Home (and by extension my Active Directory DNS) servers.

I wanted to follow up on this; the screenshot I had uploaded didn't have my laptop attempting to gain an IP address, but another screenshot did. I selected the wrong one.
I have made further progress on being able to get an IP address.

The issue I have run into now is that while I can get an IP on that subnet, I cannot get out of the subnet to the Internet.

I can ping other devices on my LAN but cannot access their web interfaces or anything, which is exactly what I wanted for internal security for this Guest VLAN.

However, the Guest VLAN should still be able to connect to the network. Clearly I have done something wrong.

Setup as follows:
ESXi has a single Ethernet to virtual switch as "LAN", then a port group for LAN (no vlan id), then another port group with vlan id 99 named VLAN_Guest.
I assigned this port group to OPNsense VM just like I did with the LAN and WAN port groups.

In OPNsense:

• Interface > Assignment > vmx0 ESXi_VLAN_Guest_Int - Static IPv4, IPv4 10.0.199.1/24, upstream Gateway: ESXi_VLAN_Guest_Gateway
• System > Gateways > Config > New > Interface ESXi_VLAN_Guest_Int, IP Addr 10.0.199.1
• Interface > Other > VLAN > vlan01, tag 99, parent vmx0 ESXi_VLAN_guest
• Firewall > NAT > Outbound > Interface ESXi_VLAN_Guest_Int, source Any, dest Any, NAT Address Interface Address
• Firewall > Rules > ESXi_VLAN_Guest_Int > Rule Protocol IPv4 Any, Source Any, Dest Any, Gateway default

Any ideas on what I'm doing wrong that would prevent devices on the VLAN getting to WAN/Internet?

Update:
In reply2 I posted my current state of affairs for getting VLAN's to work properly on my network.

Setup as follows:
ESXi has a single Ethernet to virtual switch as "LAN", then a port group for LAN (no vlan id), then another port group with vlan id 99 named VLAN_Guest.
I assigned this port group to OPNsense VM just like I did with the LAN and WAN port groups.

In OPNsense:

• Interface > Assignment > vmx0 ESXi_VLAN_Guest_Int - Static IPv4, IPv4 10.0.199.1/24, upstream Gateway: ESXi_VLAN_Guest_Gateway
• System > Gateways > Config > New > Interface ESXi_VLAN_Guest_Int, IP Addr 10.0.199.1
• Interface > Other > VLAN > vlan01, tag 99, parent vmx0 ESXi_VLAN_guest
• Firewall > NAT > Outbound > Interface ESXi_VLAN_Guest_Int, source Any, dest Any, NAT Address Interface Address
• Firewall > Rules > ESXi_VLAN_Guest_Int > Rule Protocol IPv4 Any, Source Any, Dest Any, Gateway default

Hello,
I've searched the forums and I haven't found anything that quite describes the issue I'm having. I'm also not sure whether this is an issue with the ISC DHCPv4 or if this is an issue with my TP-Link Omada SG2428P switch.

ESXi has a network port as LAN. I created a Port Group [VLAN_Guest] with VLAN ID 99. I have assigned a new network interface to OPNsense as VLAN_Guest.

In OPNsense, I have assigned this new interface (vmx0) as [ESXi_VLAN_Guest].
I have ISC DHCPv4 [ESXi_VLAN_Guest] network with DHCP setup with the subnet 10.0.199.1/24, with the range 10.0.199.40-10.0.199.254.

In TP-Link Omada controller software, I have a LAN Interface "Guest" that is enabled on the switch. I have updated a desired port with the port profile "Guest".

I plug in my laptop to this port with the guest profile. When I check out ipconfig /all I see that the interface has shown me the DNS servers, but the gateway is blank, and the IP address is in the 169.254 range indicating it's not receiving an IP.

When I check the DHCP logs, I see that the laptop does a DHCPDISCOVER, DHCP does a DHCPOFFER, then a few seconds later this repeats 2 more times...
DHCP does not "Abandon" this IP address offer, and the lapptop does not obtain a network connection.

In addition, I see that my Omada switch keeps sending DHCP requests to the DHCP server on only this guest VLAN, through this loop: DHCPDISCOVER, DHCPOFFER, DHCPREQUEST, DHCPACK, Abandons the IP address: declined, and then DHCPDECLINE. It does this through the entire IP range that is assigned and once an IP address is finally released, it attempts again to request it.

I think one of the things that makes me wonder what I did wrong is that the interface used in the DHCP lease list is LAN instead of ESXi_VLAN_Guest...

I am unsure whether this is an issue with the Switch or if I have misconfigured my DHCP server on OPNsense.

Glad I'm not the only one.

I upgraded to 21.7.1 through live upgrade, and also through a new install but having issues where I'm not able to get anything passed through. Nothing in 80, 443, 1194, and 1195. Nothing.

I think filling out an issue will help. As for now, I'm going to revert to an older version. Thank God I have a working snapshot.

So a new development has occurred. I went and setup a new ESXi VM and installed 21.1

I had taken a snapshot of my OPNsense VM prior to the upgrade and once this behavior was discovered I rolled it back to the previous snapshot (21.1.x) and the functionality was working as normal.

I attempted to utilize opnsense-revert but it appears that only works for specific packages and not the entire release. I then installed previous versions starting at 21.1, then auto-upgraded to 21.1.9_1-amd64. Found that I was still able to utilize the port forwarding correctly.
When I upgraded to 21.7 (no minor version) I was able to confirm that I was still able to utilize the port forwarding as expected. I took another VM snapshot at this point.

I then ran the upgrade and it said it was going to upgrade to 21.7.2. However, once you actually proceed with the install, there is a part where it shows "Installed packages to be UPGRADED:" and it shows that opnsense: 21.7 -> 21.7.3_3
So it appears as though the information in the update is partially wrong as well.  It did, in fact, upgrade to 21.7.3_3.

Finally, and I'm not sure how... but I have confirmed that 21.7.3_3 is working with port forwarding... for 5 minutes... maybe. Then, as I started writing up this information to you, I kept testing it ... and it started timing out.
I rolled back to 21.7 (with no configurations other than the port forwarding, and it still continues to time out. This is happening from multiple devices and I'm positive that my DNS is working properly as nothing has changed there.

I rolled back to 21.1.9_1 and for some reason it's still not working properly.

I ended up just moving to 21.1.9_1 fresh install and am going to sit with that for a while until we can confirm everything stays working in newer versions.


I know this isn't a great solution, but something you may want to consider.

Glad I'm not the only one.

I upgraded to 21.7.1 through live upgrade, and also through a new install but having issues where I'm not able to get anything passed through. Nothing in 80, 443, 1194, and 1195. Nothing.

I think filling out an issue will help. As for now, I'm going to revert to an older version. Thank God I have a working snapshot.