Azure deployment and basic routing

[msowka]

Going to summarize the OPNsense Azure deployment, and ask a trivial question.

I've bootstrapped 19.1.0 (updated to 19.1.4) on Azure using Microsoft built FreeBSD 11.2, with two intefaces:
 - LAN hn0 interface connected via DHCP to 'default' subnet 10.0.0.0/16
 - WAN hn1 interface connected via DHCP to 'ext' subnet 168.10.0.0/16

(Recall that on Azure you use Azure DHCP or the highway!)

Firstly, the order the NICs are assigned in the Azure VM and the order they're enumerated by OS is following some form of convention, ie LAN as the primary, WAN + OPT as the gateways out, ? With this ordering of intefaces, LAN hn0 DHCP having found a default gw at 10.0.0.1, OPNsense will accept that as the system default gw as show it accordingly in System>Routes>Status.

This is of course bogus, compliments of Azure pseudo-networking (SDN), as the systems default gw should be that fetched by the WAN hn1 interface.

I've tried to find ways to override gateway settings in the Interfaces>[LAN]/[WAN], and also in the System>Gateways>Single pane where I've disabled LAN_DHCP as gateway. I do understand that the Gateways that appear in the latter are part of the configuration of OPNsense services, but I'm just stabbing in the dark in having WAN hn1 recognized as the default system gw.

*** I believe I could simply reassign the NICs effectively swapping to WAN -> hn0, and LAN -> hn1, but I want to better understand the OPNsense system, and the correct way of setting this up regarless of iface enumeration.

Again, this is bogus fundamentally, and then affects basic services like Outbound NAT where LAN is used as the default Outbound NAT interface. I've made some progress in wrangling a working NAT setup on this Azure vnet, #1 with the required Azure User Defined Routes (UDRs), and then setting the Outbound NAT manually... but there must be a correct way about this.

QUESTION: What is the correct way of setting the system default gw?

Look forward to being schooled in basic OPNsense 'sense', and swapping insights re cloud deployments.

Mike

[mitsos]

The order of nics (as seen by the hardware/OPNSense) doesn't matter, I'm running numerous setups where the WAN isn't the first detected interface.

There is only one way to set up the default gateway in a DHCP scenario: don't do anything.

The biggest question is why is your LAN listening for a DHCP answer? Set it up manually with an IP in your assigned subnet and leave it at that, shouldn't require DHCP. It doesn't make any sense from a network setup perspective: Wait for a gateway to be assigned + assign the gateway received, on an internal interface?

Before you say "that's how it's supposed to work", it's not how it's supposed to work. In cloud deployments, the internal interfaces **can** but **should not** be using DHCP. DHCP is there in case you aren't running a router (which you obviously are), so addresses can be assigned by the infrastructure behind the "cloud", so your VMs can actually talk to one another. Even then, a gateway isn't needed. A VM with IP 10.0.0.1 wishing to talk to 10.0.0.2 sends out an arp request (ie "hey, anyone seen 10.0.0.2?") and the VM with that IP will answer "over here".

TL;DR: switch your LAN to manually assigned, making sure you aren't conflicting with "their" assigned gateway (ie don't use 10.0.0.1 if their gateway is 10.0.0.1). AFAIK, even that can work if the infrastructure allows you to "forget" DHCP (ie Openstack does this, personal experience).

[msowka]

deZillium, I'm with you, order of the iface should not matter, but this is Azure we're talking about And the culprit in this case is the azure vm agent preinstalled on the freebsd image. Among all the magical things it does, it manages the route table on the host, and having picked up the Azure NIC 1 / hn0, it had decided that was my default gw. For all the magical things it does, good and bad*, for the moment I will live with it rather than gutting it out.

I managed my setup by swapping the virtual NICs across the subnets, and with hn0 now assigned to WAN everything worked as expected.

My networking-foo level is intermediate, but I'm not sure if that's a hindrance or an advantage when working with unconventional networking environments like Azure SDN. As far as I can tell, DHCP is the only practical way for ip configuration in an Azure vnet, as this is how Azure fabric knows how to route the traffic. (Unpractical ways involve using user defined routes to tell azure how to reach the host that happens to be listening to 192.168.1.1 by default; yes I managed this ).

Yes again I'm with you, in a conventional networking sense I am running a router, and "of course" should be responsible for running DHCP on my LAN. But all the Azure documentation points to the fact that this is not supported, e.g. some of many:
- https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-faq#what-protocols-can-i-use-within-vnets
- https://support.microsoft.com/en-us/help/2721672/microsoft-server-software-support-for-microsoft-azure-virtual-machines (read down to section on "The following roles are not supported on Microsoft Azure virtual machines:"

deZillium, this is not to rebuff your knowledgeable suggestions, THANK YOU, but more to vent my Azure growing-pains. And yes, as per your suggestion, next time I am faced with this I will see what happens if I play with static ip assignment on the host (opnsense config), using the Azure fabric assigned ip.

Cheers, Mike

* The vm agent at this development stage is useful for basic host monitoring, password recovery, and my favorite I'm not willing to give up just yet: the Azure VM Serial console.

[mitsos]

The documentation doesn't specifically cancel anything I said: The DHCP offered by Azure is for setting up routes automatically to your VMs, in case you are going through the "private" lan side out to the internet through them (azure).

Whether or not they "block" DHCP, I doubt it, but I'm assuming deducing from the documentation that it is to avoid a DHCP conflict on the interface (ie two DHCP servers answering, which is bad unless it's an HA setup (which I have built in the past and it works just fine)).

I don't have (nor want) any experience with azure (I consider (in my opinion) any Microsoft product to be cancerous), but other SDNs that I worked with (openstack, onapp) allow you to do pretty much anything you want with the interfaces/IP assignments. I'm running a cluster spread over 3 different datacenters with a private network through openstack right this moment, something the azure documentation specifically says azure can't do.

Try it with manual assignments next time (realistically, you don't need DHCP on VMs' private networks anyway), just to test how incompetent feature-lacking azure is.

[msowka]

Yes, agreed, probably be able to run DHCP on a vnet, but again this would not be practical IMO. Yes, in a bind I can do static ip configuration assuming I get the network right, or insert contrived UDRs; but long term may run into ip address contention as someone else deploys a vm provisioned by default with DHCP.

For now, simply provisioning with WAN as my first Azure NIC does the trick.

Hazmat suit on , working on Azure,
Mike