Swap physical interfaces

[mokaz]

Hi there Team,

I recently wanted to do the following's on my OPNsense system:

• add a new physical interface
• swap my WAN interface with the newly added vtnet5 adapter (was vtnet1.xxx)
• update my main gateway reflecting the changes
• the old WAN interface (the vLAN as well as its parent interface) would have been moved toward a different Zone (keeping an IPsec tunnel through here)

Sadly, my testings went pretty south TBH =) -- I ended up restoring a VM backup.
One of the artifact I've seen was that my old (vLAN based) gateway configuration kept coming back in the GUI/XML configuration and traffic didn't seemed to flow through the newly assigned WAN member. DHCP client had been functioning on the newly assigned interface though.

Is there anything I need to pay attention to before attempting the shift again ?

Let me know,
Cheers,
m.

[CJ]

If you're adding an interface then normally it's relatively easy, if somewhat annoying to swap things around.

Can you post a diagram or list showing your desired before and after states?  It sounds like you've got a lot of moving parts which can add to the complexity.

[cookiemonster]

For Proxmox at least, don't forget that after adding a device that uses pcie lanes like nvme, hba or nics, the order of them changes and VMs need to be reconfigured to reflect.
What I've learned to do is disable autostart of VMs before shutting down. Then add hardware. Then boot and compare VM config device ids against the what shows in the console.

[mokaz]

If you're adding an interface then normally it's relatively easy, if somewhat annoying to swap things around.

Can you post a diagram or list showing your desired before and after states?  It sounds like you've got a lot of moving parts which can add to the complexity.

Thanks a lot for your help guys. Hence, I gave this a 2nd shot and now everything has been an home run from start to finish...

Basically to explain you what/why I wanted to change this scheme -- historically I've had all the traffic of that OPNsense (was something else before/big up for OPNsense, way better...) passing through another main NGFW. That main NGFW had all the needed objects/policies enabled to reach potential services behind the OPNsense... This was to simplify the design at the time (single WAN uplink).. Although, that main NGFW is somewhat of a personal playground which means that conducting maintenance on that one would had disrupted the OPNsense box connectivity -- fact which I wanted to change because newly so, there are now peoples behind the OPNsense box, whooohooo...

Well the problem here most likely lied in between the chair and the keyboard =)

Thanks a lot for your help,
Cheers,
m.

[CJ]

For Proxmox at least, don't forget that after adding a device that uses pcie lanes like nvme, hba or nics, the order of them changes and VMs need to be reconfigured to reflect.
What I've learned to do is disable autostart of VMs before shutting down. Then add hardware. Then boot and compare VM config device ids against the what shows in the console.

Fun.  I've only just started playing with proxmox and haven't done any pcie passthrough yet.

Basically to explain you what/why I wanted to change this scheme -- historically I've had all the traffic of that OPNsense (was something else before/big up for OPNsense, way better...) passing through another main NGFW. That main NGFW had all the needed objects/policies enabled to reach potential services behind the OPNsense... This was to simplify the design at the time (single WAN uplink).. Although, that main NGFW is somewhat of a personal playground which means that conducting maintenance on that one would had disrupted the OPNsense box connectivity -- fact which I wanted to change because newly so, there are now peoples behind the OPNsense box, whooohooo...

This is one of the reasons I like having a physical OPNsense.  I don't have to take down the internet in order to update servers, etc.  But I totally understand what you mean about it becoming a critical piece of infrastructure.  That's why I always make sure I have plenty of time available to do update and upgrades.  They're usually painless and quick, but sometimes you end up like this.  https://xkcd.com/349/

Well the problem here most likely lied in between the chair and the keyboard =)

Thanks a lot for your help,
Cheers,
m.

I find it usually does as hardware today is much more solid and reliable than it used to be.

[cookiemonster]

For Proxmox at least, don't forget that after adding a device that uses pcie lanes like nvme, hba or nics, the order of them changes and VMs need to be reconfigured to reflect.
What I've learned to do is disable autostart of VMs before shutting down. Then add hardware. Then boot and compare VM config device ids against the what shows in the console.

Fun.  I've only just started playing with proxmox and haven't done any pcie passthrough yet.

It is good but there are some gotchas like this one that can bite. Some hardware is better than others for pcie passthrough. You'll soon need to start reading about iommu groups that some motherboards are better for it.

[CJ]

It is good but there are some gotchas like this one that can bite. Some hardware is better than others for pcie passthrough. You'll soon need to start reading about iommu groups that some motherboards are better for it.

I'm mainly planning on using proxmox because it provides an easier remote screen experience than xcpng.  About the only things I'm looking at passing through into any VMs are usb dongles for HomeAssistant.