TCP connections fail after ISP IP renewal

[gorillaporcupine]

Hi, I have a weird problem:

I am using dyndns to host some services at home. If my ISP is providing me with a new public ip address, all is synced correctly vis ddclient. DNS is resolving the correct address and also wireshark VPN (UDP!) works fine. But all requests to my piblic IP based on TCP are dropped. Or at least my self hosted websites cant be resolved.
If I reload my PPPoE connection till I get a new IP from the ISP (3 times normally), TCP traffic to my public IP is resolved correctly again. This is a very inconvenient issue, because I need to monitor my service constantly and if it's down, I need to log in via VPN to reload the PPPoE Interface till I get a new IP.

Any ideas how to debug this further? I contacted the ISP and also tried a different router. No problems there. That's why I assume it is something buggy in OPNsense...

[CJ]

What are you using to check your public IP?

Are you saying that UDP works via the dynamic domain but not TCP?

What do you mean by rebooting to get a new IP?  You had x.x.x.1, it changed to x.x.x.2, and then you rebooted until you got x.x.x.3?

[gorillaporcupine]

I use the ddclient plugin, it checks for a changed IP on the WAN interface every 5min. I checked the DNS resolving and wireguard via mobile broadband on my smartphone and from a different location.

Yes, UWP works. Wireguard VPN is not affected by the ISP side IP change. My website does not resolve though. Also, the DNS system points to the correct IP after changed by ISP. So dynamic dns works, too.

I reload the PPPoE Interface. Normally 3 times reload and the ISP provides a new IP. Afterwards all is fine and my website is loading.

[CJ]

Okay, I'm still a bit unclear about the situation which is why I was trying to get exact details about the steps you're performing. Please answer all of the following questions so I can understand where the problem is occurring.  BTW, are you aware that ddclient is going away and you should be using the native backend?
 
Initial working situation:
WAN interface reports 198.51.100.12.  DNS lookup of dyn.example.com resolves to 198.51.100.12.  Wireguard connects and local webserver works correctly from externally.

ISP lease expires and provides new lease.  WAN interface reports 198.51.100.15.  DNS lookup of dyn.example.com resolves to 198.51.100.15?  Wireguard existing tunnel continues working?  Wireguard new connection works?  Local webserver receives no incoming connections?  OPNSense forwarding rule receives no incoming connections?

WAN down/up cycle until new lease received.  WAN interface reports 198.51.100.25.  DNS lookup of dyn.example.com resolves to 198.51.100.25?    Wireguard existing tunnel continues working?  Wireguard new connection works?  Local webserver receives incoming connections?  OPNSense forwarding rule receives incoming connections?

[gorillaporcupine]

First of all, thank you for your time and effort.

I will try to clarify as good as possible:

Initial working situation:
WAN interface reports 198.51.100.12.  DNS lookup of dyn.example.com resolves to 198.51.100.12.  Wireguard connects and local webserver works correctly from externally.

correct, this is the working situation.

ISP lease expires and provides new lease.  WAN interface reports 198.51.100.15.  DNS lookup of dyn.example.com resolves to 198.51.100.15?  Wireguard existing tunnel continues working?  Wireguard new connection works?  Local webserver receives no incoming connections?  OPNSense forwarding rule receives no incoming connections?

Correct. Wireguard stops working at renewal and is working a couple of minutes later, when DNS is updated. Which is the expected behavior. Connections to my webserver are refused.
I will need to look into the connections, when the issue occures again. If I recall correctly, opnsense logged incoming connections but I need to verify this.

I am aware of the deprecation of ddclient, I had an issue with the OPNsense backend to not detect an IP change. It's in my backlog to look into this.

[meyergru]

Did you see the question marks in CJ's request?

Connections to my webserver are refused.

This does not answer the question what part of the connection does not work. If DNS is not being updated at all or if it is being cached, you probably reach the old IP.

It is also a question of how you test this: e.g., current browsers use DoH or DoT, not DNS over UDP.

Site-to-site Wireguard on the other hand is often configured to be able to bring up the connection from either side, such that a change in one side is healed more quickly even when DNS is slow.

I doubt that suddenly, OpnSense chooses to deny new TCP connections unless your firewall rules are weird. After all, each incoming connection has to be a new one after a WAN IP renewal - all old connections must have been dropped already.

[gorillaporcupine]

Thank you for your feedback. Please find all questionmarks and my answers below.

ISP lease expires and provides new lease.  WAN interface reports 198.51.100.15.  DNS lookup of dyn.example.com resolves to 198.51.100.15?

- yes. Tested with external service dnschecker.org, if the IP is resolved correctly.

Wireguard existing tunnel continues working?

no. The tunnel is terminated after ISP renewal.

Wireguard new connection works?

yes. peer to server connection. no site to site. WG connects to the new IP.

Local webserver receives no incoming connections?

correct. also using the new wan IP does not reach the web server.

OPNSense forwarding rule receives no incoming connections?

to be verified as soon as the issue occures again (mostly every couple of days)

I doubt that suddenly, OpnSense chooses to deny new TCP connections unless your firewall rules are weird.

me too, and thats why I am asking for help here. I need help to debug this. Again, reloading the interface untill I get a new IP, solves the issue without changes to any of my other settings.

After all, each incoming connection has to be a new one after a WAN IP renewal - all old connections must have been dropped already.

I am aware of that.

[cookiemonster]

Hi, I have a weird problem:

I am using dyndns to host some services at home. If my ISP is providing me with a new public ip address, all is synced correctly vis ddclient. DNS is resolving the correct address and also wireshark VPN (UDP!) works fine. But all requests to my piblic IP based on TCP are dropped. Or at least my self hosted websites cant be resolved.
If I reload my PPPoE connection till I get a new IP from the ISP (3 times normally), TCP traffic to my public IP is resolved correctly again. This is a very inconvenient issue, because I need to monitor my service constantly and if it's down, I need to log in via VPN to reload the PPPoE Interface till I get a new IP.

Any ideas how to debug this further? I contacted the ISP and also tried a different router. No problems there. That's why I assume it is something buggy in OPNsense...

Some terminology clarity would be useful.
- "But all requests to my piblic IP based on TCP are dropped" . Are they being dropped by the firewall and can be seen from the OPN side? Or do you mean connections from another network (mobile phone for instance) time out, get rejected, what? And are they done to the new ip or to an url?
- "Or at least my self hosted websites cant be resolved.". Which one is it, dns is updated or not? Done an external dns query? Resolves or not to the new ip?
These websites, how are they being served through OPN, haproxy, port forwarding, nginx, etc?

[CJ]

Local webserver receives no incoming connections?

correct. also using the new wan IP does not reach the web server.

OPNSense forwarding rule receives no incoming connections?

to be verified as soon as the issue occures again (mostly every couple of days)

Is this a straight port forward or a proxy?  As CM mentioned, more details about the firewall rules and hosting setup would be helpful.

[gorillaporcupine]

Hi, thank you again for your patience and effort. I will first answer the new questions and afterwards give a detailed description of my hosting setup.

Some terminology clarity would be useful.
- "But all requests to my piblic IP based on TCP are dropped" . Are they being dropped by the firewall and can be seen from the OPN side?

To be verified as soon as the issue occurs again.

Or do you mean connections from another network (mobile phone for instance) time out, get rejected, what?

They time out.

And are they done to the new ip or to an url?

Both. The URL (resolving the correct IP via DNS) leads to a timeout as does the connection using the new IP.

- "Or at least my self hosted websites cant be resolved.". Which one is it, dns is updated or not?

https://nextcloud.baltic-hosting.de DNS is updated correctly

Done an external dns query? Resolves or not to the new ip?

yes, using dnschecker.org. It resolves to the new IP.

These websites, how are they being served through OPN, haproxy, port forwarding, nginx, etc?

I use port forwarding to my nginx proxy, running in a VLAN. The webserver hosting the service is in a different VLAN.

My hosting setup:

NextCloud | Proxy | Port Forwarding | Internet

The webserver, hosting the NextCloud service, is in VLAN "Management". There are no restrictive rules in this VLAN. Connections are allowed any to any.

The Proxy (Nginx) is in the VLAN "DMZ" with allow rules to reach my NextCloud webserver and the other services I am hosting. All other local addresses are denied. Internet access is available for updates.

Port forwarding is used to expose the Proxy and the necessary ports (80, 443).
Port forwarding is also exposing the Wireguard VPN Port to the internet. I am using the Wireguard plugin of OPNsense.
Please find screenshots of the settings attached.

[CJ]

It sounds like we're at a standstill until you can verify which part of the chain is failing.  Make sure you have logging turned on so that you can see where it fails.

Where are you hosting the proxy and nextcloud?  On a separate machine or OPNSense?

[gorillaporcupine]

I have logging turned on for all nat rules. It can only be a matter of days till the next IP change.

Opnsense is a VM on a Proxmox host. NextCloud and Proxy are LXC containers.

Regards and I keep my fingers crossed ISP is changing the IP soon 

[newsense]

Unplugging the modem might suffice

[CJ]

Unplugging the modem might suffice

Only if they do the cable and not the power.  Power loss probably won't replicate the situation correctly.

[gorillaporcupine]

Actually, there is no modem. I have FTTH, terminating in an SFP module in my servers NIC.

Referring to CJs post: In the meantime I updated to OPNsense 23.7.5. I also rebooted my server due to BIOS and Proxmox OS updates. This lead to new IP addresses from ISP but all services remained reachable after IP change. DNS was also updated correctly by Dynamic DNS.

I have external monitoring up and running. As soon as I lose connection to my website again, I will dive into the logs and post them here.