Messages

3f: that's how it's supposed to be

3e is interesting: The server that answered your query was 127.0.0.53. That's the local subnet. I thought you changed the DNS servers to PIA's ones. Can you double check the DHCP DNS settings + the DNS settings for opnsense?

Before you double check the servers, from your LAN: "dig www.google.com @{your actual dns server or 1.1.1.1}" ie "dig www.google.com @1.1.1.1". I'm betting a banana that it will answer correctly.

The documentation doesn't specifically cancel anything I said: The DHCP offered by Azure is for setting up routes automatically to your VMs, in case you are going through the "private" lan side out to the internet through them (azure).

Whether or not they "block" DHCP, I doubt it, but I'm assuming deducing from the documentation that it is to avoid a DHCP conflict on the interface (ie two DHCP servers answering, which is bad unless it's an HA setup (which I have built in the past and it works just fine)).

I don't have (nor want) any experience with azure (I consider (in my opinion) any Microsoft product to be cancerous), but other SDNs that I worked with (openstack, onapp) allow you to do pretty much anything you want with the interfaces/IP assignments. I'm running a cluster spread over 3 different datacenters with a private network through openstack right this moment, something the azure documentation specifically says azure can't do.

Try it with manual assignments next time (realistically, you don't need DHCP on VMs' private networks anyway), just to test how incompetent feature-lacking azure is.

The order of nics (as seen by the hardware/OPNSense) doesn't matter, I'm running numerous setups where the WAN isn't the first detected interface.

There is only one way to set up the default gateway in a DHCP scenario: don't do anything.

The biggest question is why is your LAN listening for a DHCP answer? Set it up manually with an IP in your assigned subnet and leave it at that, shouldn't require DHCP. It doesn't make any sense from a network setup perspective: Wait for a gateway to be assigned + assign the gateway received, on an internal interface?

Before you say "that's how it's supposed to work", it's not how it's supposed to work. In cloud deployments, the internal interfaces **can** but **should not** be using DHCP. DHCP is there in case you aren't running a router (which you obviously are), so addresses can be assigned by the infrastructure behind the "cloud", so your VMs can actually talk to one another. Even then, a gateway isn't needed. A VM with IP 10.0.0.1 wishing to talk to 10.0.0.2 sends out an arp request (ie "hey, anyone seen 10.0.0.2?") and the VM with that IP will answer "over here".

TL;DR: switch your LAN to manually assigned, making sure you aren't conflicting with "their" assigned gateway (ie don't use 10.0.0.1 if their gateway is 10.0.0.1). AFAIK, even that can work if the infrastructure allows you to "forget" DHCP (ie Openstack does this, personal experience).

You are changing too many things and I can't keep track. Doing this without being in front of the screen to properly troubleshoot it is already confusing.

Recap:
1) Set up your VPN tunnel (check that auto NAT is enabled)
2) Change the "LAN to any" allow rule to gateway > VPN.
3) Then actually post the output for:
3a) ping from your LAN to your OPNSense
3b) ping from your LAN to the VPN gateway (the PIA server's VPN IP)
3c) ping from your LAN to 8.8.8.8
3d) ping from your LAN to www.google.com
3e) On OPNSense Interfaces > Diagnostics > DNS lookup for www.google.com
3f) You mentioned linux, so I'm assuming that you are using linux instead. Do a "dig www.google.com" from your LAN.
3g) Post your OPNSense routes when the tunnel is established (System >  Routes> Configuration), redact if necessary.

"no change on their end as far as they can tell", fixed that for them  ;)

I refuse to believe your router suddenly decided to stop working without anything changing. In my 20+ years of configuring networks, I've never seen this happening. It doesn't matter if you get a dozen or a hundred IPs, compared to the hundreds of thousands your ISP is handing out, that's merely a statistical error.

Every ISP worth its salt should be able to "run a trace" (packet capture) on your connection for a few hours (usually 24 hours) to see why it stops working. Have them run this on their end.

Updated 4 qotoms so far, mixed setups (bridged modems, routing modems, PPPoE), no issues.

When the WAN shows down, is OPNSense showing any IP on your WAN? Is your modem in bridged or in routing mode?

I'm betting a penny on an IP conflict in your ISP's network. Sounds like "it forgot" that it assigned that IP and assigns it to another customer a bit later on, which obviously is a dumpster fire.

Verify that ping is indeed going through the tunnel by capturing some traffic (interfaces > diagnostics > packet capture > your VPN interface) while pinging from your LAN.

If pinging with an IP works but not with the hostname, it's a DNS issue. Is the ahole, pardon pihole, on a different interface other than LAN by any chance?

I'm assuming you are using windows, so do an "nslookup www.google.com" from a command line when it's not working.

You only need a port forwarding rule for that, you don't need haproxy (unless the "endpoint" refuses to talk to anything outside of its network **cough** HP switches **cough**).

Get rid of haproxy and any additional IPs you have added. Add a port forwarding rule, interface WAN, source any, destination any, port (the port you want), internal IP (the internal IP for the controller you are trying to connect to). Hint: set up a port alias if you need more than one ports and use that alias in the rule instead of a port.

Sidenote: since IoT devices aren't exactly on the cutting edge of security, make sure that that controller is on its own interface (VLAN) without any access to the rest of your network, because when the day comes that it will be added to a botnet (and the day **will** come eventually), you don't want that threat in your network.

I would personally set up a VPN and access the controller that way, instead of opening access to it to the world, but that's just me. This post is for educational purposes only, don't blame me when the controller eventually gets compromised  ;)

Are you sure that I need to add that gateway manually?  There is already one there labeled  PENVPNCLIENT_VPNV4.  Besides, how do I automate retrieval of that IP address?  Surely, it does not need to be added by hand each time?  At any rate, I added a gateway manually using the IP address of the VPN server, and made it the gateway for the LAN rule, but no dice.

1) Where did that gateway come from?
2) I'm willing to bet you used the public IP of the VPN server instead of the tunnel IP.

Not sure about where the last rule goes.  Well, I did the above.  The VPN logs shows that the "Initialization sequence is completed".  So the OpenVPN client is communicating fine with the
server.  From the OPNsense box, I can ping the outside world.  However, I then have no internet access from th LAN.  Only, if I change the gateway for "default allow LAN to any" rule back to default, do I get access, but not through the VPN.

The OPNSense box doesn't go through the VPN tunnel, that's why you can ping outside. Can you ping the VPN gateway (the remote server's tunnel IP, not the public IP) from the LAN?

WRT the DNS: your setup is a disaster, let's get the VPN fixed first then we'll worry about the DNS.

What hardware are you running this on?

The screenshot almost made my mind blow up. You are connecting through your DDNS, which means everything is working up to that point: DDNS got updated, forwarding rules work (I'm pretending I didn't see you open up your firewall's remote admin to the world), but the system looks barely functional.

Since you said you've seen this a couple of times, I would say that it's either a bad RAM stick or bad HDD/SSD (SSD should have reverted to a read only state if it failed, could explain why half of the things are working and half aren't).

I would start with a hardware diagnostic first, your hardware looks haunted  ;)

Because you had one too many beers and forgot that you edited your WAN-forward rules?  ;D

The WAN rules are listening for a private IP as the destination in order to trigger and since I see the block private on top, that means you aren't in a double nat scenario, so the rules are listening on the wrong IP, hence never triggered.

You are welcome, send some of that beer my way  ;D

RFC4632: Do not use "class" to describe subnets (obsoleted 13 years ago): https://tools.ietf.org/html/rfc4632#section-3

That being said:
1) Your haproxy listen IP (public service) is wrong, it should be listening on you external IP
2) Why are you using haproxy to forward ports? If it's a single server, just forward the port directly (under firewall)

Now that we fixed the modem access, on to fix the VPN:

Modify your "LAN to outside rule" so that the gateway used (there is a gateway option, you need to add the gateway's IP manually under  System > Gateways > Single) is the remote server's VPN address (the one you get *after* establishing the tunnel). That will force outgoing packets to go through that particular gateway and return packets (ie answers to *your* packets will have no other way but to flow through the VPN).

Make sure you don't get locked out. Check (and recheck) that there is a rule that allows OPNSense webgui access **BEFORE** your modified rule.

That modified gateway rule will direct everything over the tunnel. Assuming your provider (I see PIA mentioned there, so I'm going with private internet access) allows you to query DNS over the tunnel (it should) then the only thing left is to modify your DHCP server settings so that the DNS it gives is the PIA one and/or modify OPNSense's DNS (you really really should be using unbound in "full resolver" mode + have that go through the tunnel) and you have completed the circle. For unbound, create a rule to allow traffic from the firewall itself to the world + VPN gateway as before.

Feel free to adjust/add rules as you see fit for access. Also make sure that there is a rule that allows traffic from your LAN to your modem's subnet **before** the VPN rules.

Enjoy, you are welcome  :)

The first rule is a) wrong and b) not needed, so delete it. it is messing up everything in your modem's subnet. That's "a" reason for not reaching your modem. Planning ahead for "[citation needed]": You are taking the source subnet and translating it to the (same) destination or in simple words you are taking an orange and translating it to an orange.

VPN is another topic, for now we are talking about reaching your modem. Offtopic: you add a default route for your VPN, with exceptions for your local subnets (ie modem) as well as your remote endpoint (so it actually knows how to reach it when the tunnel is being established/off), not NAT. VPN endpoints are "routers", they understand routing in order for your packets to get from point A to point B. What I'm trying to say is you don't use NAT on the entry, you use it on the exit side, ie the server side. I doubt the OPNSense documentation said anything about adding NAT on your entry point.

Do me a favor, select automatic NAT (the first option), then post a screenshot of your modem's interface when it connects as a thanks  ;)