upgraded to 17.7.10 - lost interface assignments

[jnm]

Last night as I was going to bed, I fired off the update and reboot to 17.7.10 on my APU. I got up this morning to a down network. After dragging out the laptop and serial adapter, I found the firewall was up but had reverted all the interface assignments to defaults (re0 -> WAN; re1 -> LAN; re2 -> unconfigured) and had garbled the IP address scheme.

So we went from:

• re0 -> Public -> <private /24 network 0> (static/t6)
• re1 -> Private -> <private /24 network 1>(static/t6)
• re2 -> WAN -> DHCP(v6)
• zt* -> ZeroTier -> <private /24 network 2>

to:

• re0 -> WAN -> <private /24 network 1 from above> (static/t6) 
• re1 -> LAN -> unconfigured
• re2 -> unassigned -> unconfigured
• zt* -> unassigned -> unconfigured

Once I reassigned interfaces and reset the IP schemes, so far it looks like everything else is okay.
I don't have any idea what happened, but ain't nobody got time for that.  If there's an upgrade log somewhere y'all want me to open a Github issue against, point me to the path, and I'll get on it. Or if there's more specific information you'd like, I'll see what I can do.

[jnm]

Suricata was down, too. I have it listening on all interfaces, and I had to remove and re-add the Public interface to get it to start. Presumably has something to do with having to reassign/reconfigure OPT1/Public.

[jnm]

Aaaand it garbled my Web Proxy settings (interfaces and access control network segments). Neat.

[franco]

Please lock your Zerotier interfaces in their respective interface settings (second checkbox). They are not to be considered stable interfaces. It can happen.


Cheers,
Franco

[jnm]

Please lock your Zerotier interfaces in their respective interface settings (second checkbox). They are not to be considered stable interfaces. It can happen.


Cheers,
Franco

Frankly, if it was just ZeroTier, it'd be no big deal. Even breaking Suricata is something relatively trivial because it's "just" my home network, and nothing critical depends on IDS/IPS. Reassigning WAN+LAN and breaking Squid, though… that's huge, because it means I have no network.

[franco]

Please lock your Zerotier interfaces and this will not happen.


Cheers,
Franco

[jnm]

Please lock your Zerotier interfaces and this will not happen.


Cheers,
Franco

1) Do I understand you to be saying that an unlocked ZeroTier interface can cause an OPNsense upgrade to remove all interface configuration and assignment?
2) In what use case would an administrator want unstable interfaces? Or, put another way, if it's possible for some operation to cause instability in the interface assignments, why not lock them by default?

[franco]

1) Zerotier is a plugin maintained by an external contributor. It may not have the perfect integration in terms of e.g. OpenVPN stability. It is a known issue. It also needs work in the core, but it is what it is.

https://github.com/opnsense/plugins/issues/239

2) Nobody "wants" unstable interfaces. How do you propose to lock them by default? How would a config migration from another hardware to a newer system work where interface names change? That's two of the challenges in making the experience less rocky in the future. Any help here is highly appreciated.


Cheers,
Franco

[jnm]

1) Zerotier is a plugin maintained by an external contributor. It may not have the perfect integration in terms of e.g. OpenVPN stability. It is a known issue. It also needs work in the core, but it is what it is.

https://github.com/opnsense/plugins/issues/239

2) Nobody "wants" unstable interfaces. How do you propose to lock them by default? How would a config migration from another hardware to a newer system work where interface names change? That's two of the challenges in making the experience less rocky in the future. Any help here is highly appreciated.


Cheers,
Franco

Those are reasonable responses, Franco; thank you. I didn't understand why that checkbox was there, because it seemed to me that interface assignments should be considered stable. Lesson learned; you can be sure I've locked them all now.

I don't know what checking that box does on the backend when you lock the interface and I'm not a PHP dev, so I can't offer any suggestions for how to lock one by default. I can say that I have a greater expectation of configuration stability through on-device upgrades and reboots than I would for migrations to dissimilar hardware, so if I were prioritizing development efforts, I'd look toward the former before worrying much about the latter. We deal in closed-source firewalls (one of which has a free home version that can be installed on off-the-shelf Intel hardware) at my day job, and I came to OPNsense by way of a solid 5-year run with pfSense. This is the first platform I've seen that didn't treat interface assignments as stable.

I know y'all are working hard, Franco, and I do appreciate it. Keep it up; it's a worthwhile effort.

[franco]

Hi jnm,

Those are reasonable responses, Franco; thank you. I didn't understand why that checkbox was there, because it seemed to me that interface assignments should be considered stable. Lesson learned; you can be sure I've locked them all now.

You say stable: you mean immutable in terms of keeping your configuration.

I say stable: I mean recoverable in case of hardware failures et al.

The issue you describe happens with two devices:

USB devices and Zerotier virtual devices. Both are latent failures and work most of the time anyway, well, until they don't.

I don't know what checking that box does on the backend when you lock the interface and I'm not a PHP dev, so I can't offer any suggestions for how to lock one by default. I can say that I have a greater expectation of configuration stability through on-device upgrades and reboots than I would for migrations to dissimilar hardware, so if I were prioritizing development efforts, I'd look toward the former before worrying much about the latter.

With the former in mind, the priority for this is hard to keep high all things considered. Everybody wants something.

But above all, everybody has a free flexible networking toolbox, also free from warranty and liability[1]. I'll try to find a way to lock Zerotier interfaces by default when they are created. But note the lock will never cover you completely from impending doom anyway and anything not related to discussing improvements or questions about how to work around such issues is not the best way to spend time on this project.


Cheers,
Franco

[1] https://github.com/opnsense/core/blob/master/LICENSE