20.7 & VMXNET3 to VMX Order

[ThyOnlySandman]

Making changes to environment for WIFI client bridge / Cell phone hotspot failover.

Adding additional VMXNET3 nics on existing VM throws entire VMXNET3 to VMX# out of whack.  Breaks entire FW.  One needs to re-assign going through manual pairing referencing MAC on VM to VMX MAC shown in Opnsense.  Assign port groups appropriately.  It works but kinda dirty - especially if not documented for the next guy.

So - my approach in past 20.1 has been OPNsense must just always have all 10 NICs added - so if future changes are required, then VMXNET3 NICs already present, thus no re-ordering on next boot.  I just use NULL Vmware Portgroup with no uplinks + not connected for unused NICs.

Just tried a a fresh 20.7 install with all 10 VMXNET3 from the start.  Same behavior of random ordering.
Reading up a little bit - appears to be an ongoing issue with more than 4 VMXNET3 that throws ordering out of whack.

Anybody have a trick that I can manually adjust for a 1:1 assignment.  I want Network adapter 1 in VM to = VMX0 , Network adapter 2 in VM = VMX1 , ect.

Thanks

[ThyOnlySandman]

Ok - Have some testing to do but believe may have got it.  VM VMX edit of pcislotnumbers.

Following this link gave me hint where to begin.  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198406

If you are adding more than 8 interfaces the mapping should look like this
PciSlot# --------------------Interfaces ----------Hex values for Pcislot#
160 1184 2208   ===>   vmx0   vmx1   vmx2   ===>   A0   4A0   8A0
192 1216 2240 ===>   vmx3   vmx4   vmx5   ===>   C0   4C0   8C0
224 1248 2272 ===>   vmx6   vmx7   vmx8   ===>   E0   4E0   8E0
256 1280 2304   ===>   vmx9   vmx10   vmx11   ===>   100   400   900

However after much trial and error I discovered that scsi0.pciSlotNumber = "160" was interfering with suggested address sequence.  VMX were still outta order...

Reviewing scsi0.pciSlotNumber I found other values for =161 which is meant for 2nd SCSI controller on VM.  Tried it but power on overwrote it to new -  scsi0.pciSlotNumber = "1280"

Have more testing and reading to do but this VMX config has all 10 VMXNET3 in proper order + test VM did boot with modified SCSI.

ESXi 7
VM Version = 15
Guest OS = FreeBSD 12 or later versions (64-bit)

scsi0.pciSlotNumber = "1280"
ethernet0.pciSlotNumber = "160"
ethernet1.pciSlotNumber = "1184"
ethernet2.pciSlotNumber = "2208"
ethernet3.pciSlotNumber = "192"
ethernet4.pciSlotNumber = "1216"
ethernet5.pciSlotNumber = "2240"
ethernet6.pciSlotNumber = "224"
ethernet7.pciSlotNumber = "1248"
ethernet8.pciSlotNumber = "2272"
ethernet9.pciSlotNumber = "256"

I read enough to be dangerous - try at own risk.

[bartjsmit]

Any mileage in assigning fixed MAC addresses to the vNIC's rather than subjecting them to the ESXi randomiser?

Bart...

[ThyOnlySandman]

Well I'm not sure about 20.7 but on 20.1 MAC didn't seem to help ordering
In 20.1 I edited the VMX so MACs all same but ending in :01, :02, 02, ect - Added all 10 so config could never change.  Then with the manually edited MACs it made it easier to cross reference for the matching up / port group assignments.

However - I am happy to report though that using same config I shared in post 2 with modified pciSlotNumber,  I migrated my main production firewall.  All seems well thus far.  And all NICs in order.  Went from from 3 VMXNET3 to 10 just to ensure nothing changes.

I did apply the VMX edits prior to booting VM which was a mistake.  Power-on completely changed them.  Then shutdown and applied again with WINSCP.  Booted - pciSlotNumber(s) changed again.  Shutdown.  Edited via Vsphere VMX editor.  pciSlotNumber edits remained and all VMXNET3 to VMX match. 

If one wants to try this I suggest powering on VM after VMNET3 adds, shutdown, then edit via VMX editor.  (Again at own risk)
 
I'll see if pciSlotNumber edits cause troubles - but so far so good.

Also with what started my interface adjustment project today - Happy to report that a free Linksys E2100 I acquired was successfully flashed with DD-WRT firmware, converted into WIFI client bridge, integrated into public switch vlans/trunking between ESXi hosts, and setup for WAN failover in Opnsense using my Verizon Cell phone hotspot.  It works!  Yaaay networking!

[ThyOnlySandman]

Well I just got my WAN failover working / tested right when I posted that.
Now not wanting all my systems to eat up my VZW data plan I'm looking at policy based routing.

Off topic of this post but hey - how does one set firewall rules for specific gateways while still using unbound DNS.  Soon as I enable my primary Inside VLANS Allow rule to use specific gateway DNS goes kaput.

Multi WAN guide here https://docs.opnsense.org/manual/how-tos/multiwan.html# says to put in DNS/Gateway config but I want Unbound DNS over TLS.  My understanding is System - General will use standard DNS.   Hmm.

EDIT:  Nevermind - just being dumb. 
All my internal DNS goes to DCs - DCs forward to Opnsense/Unbound.  Needed rule at top allowing internal DCs to hit LAN INT IP UDP 53.  Then following rule with specific Gateway specified.

[ThyOnlySandman]

Just final report here.
No issues to report with modified pcislotnumber changes.

Also made new VM template with modified pcislotnumber - converted into VM template and created new Opnsense VM from template.  (This generates all new MACs for all 10 NICs.)  The pcislotnumber changes remained within new VM and on first boot on NICs in proper order.

[Archanfel80]

This is an old probably a decade old issue. And its related to the freebsd, nothing to do with the opnsense.
Honestly freebsd getting worse and worse every year. Im not sure its a good idea using as a base for anything anymore. Opnsense have a lot of issue recently what is mostly related to the freebsd.

[ThyOnlySandman]

Yeah long standing issue - I found quite a bit of old questions to this interface mismatching issue but not much in way of solution.  Just that one post on bugs.freebsd.com.

Well - I consider this issue resolved.  Since posting I've not had one issue with pcislotnumber adjustments.

I've also made some VMware templates and transferred to another vsphere environment using ESXi 6.7 (I'm on 7) The VM template worked perfectly with all 10 interfaces match.  And because it already has 10 max interfaces - No one would ever need to add a new vmxnet3 to VM which can break interface mappings on first boot.  Removing a VMXNET3 interface would likely break it.  But no need to - just leave unused on Null networks + disconnected.

I like consistency.  So all my opnsense deploys will look like:

VM Net 1, vmx0 - Primary WAN
VM Net2, vmx1 - Secondary WAN
VM Net3, vmx2 -  CARP
VM Net4, vmx3 - Inside LAN (Small net to L3 switch)
VM Net5 , vmx4 - MGT

TTFN