Hi,
Bear with me, because this is a bit complicate:
I have two networks on two locations:
On location A, I have a network with a Unifi router, and port forwarding to some ports, including 80 and 443.
On location B, there's Opnsense and also port forwarding to some of the same ports as location A.
I have web servers and mail servers on both locations.
They are connected via a site-to-site Wireguard, which has worked flawlessly for almost two years.
Yesterday, out of the blue, I suddenly couldn't read the websites I host on location B, but _only from location B_! The world could still access them.
The webserver on location B also couldn't get any "curl" from the sites hosted at location A.
I did lots of tcpdump, and Wireshark - as well as OPNSense logs, show me a lot of SYN_SENT between the two locations on the ports that are failing (80, 443, etc.).
This is what the "States" logs show me:
all tcp LOCATIONAIP:61025 LOCATIONBIP:443 LOCATIONBSERVER:443 CLOSED:SYN_SENT
all tcp LOCATIONAIP:61025 LOCATIONBSERVER:443 SYN_SENT:CLOSED let out anything from firewall host itself
all tcp LOCATIONAIP:61029 LOCATIONBIP:443 LOCATIONBSERVER:443 CLOSED:SYN_SENT
all tcp LOCATIONAIP:61029 LOCATIONBSERVER:443 SYN_SENT:CLOSED let out anything from firewall host itself
all tcp LOCATIONAIP:61031 LOCATIONBIP:443 LOCATIONBSERVER:443 CLOSED:SYN_SENT
all tcp LOCATIONAIP:61031 LOCATIONBSERVER:443 SYN_SENT:CLOSED let out anything from firewall host itself
all tcp LOCATIONAIP:46838 LOCATIONBIP:443 LOCATIONBSERVER:443 CLOSED:SYN_SENT
all tcp LOCATIONAIP:46838 LOCATIONBSERVER:443 SYN_SENT:CLOSED let out anything from firewall host itself
all tcp LOCATIONAIP:61040 LOCATIONBIP:443 LOCATIONBSERVER:443 CLOSED:SYN_SENT
all tcp LOCATIONAIP:61040 LOCATIONBSERVER:443 SYN_SENT:CLOSED let out anything from firewall host itself
LOCATIONAIP = WAN IP location A
LOCATIONBIP = WAN IP location B
LOCATIONBSERVER = webserver on location B
I see on tcpdump that traffic does goes back and forth, but I have no idea why it is getting this SYN_SENT:
19:03:31.056683 eth1 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [none], proto TCP (6), length 64)
LOCATIONAIP.61153 > LOCATIONBIP.443: Flags \[S\], cksum 0xe3b4 (correct), seq 973490560, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2633606817 ecr 0,sackOK,eol], length 0
19:03:31.106573 eth0 In IP2 (invalid)
19:03:31.106573 eth0.21 In IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto TCP (6), length 64)
LAPTOPIP.61154 > LOCATIONBIP.443: Flags [S], cksum 0x4518 (correct), seq 697730383, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 552186860 ecr 0,sackOK,eol], length 0
19:03:31.106995 eth1 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [none], proto TCP (6), length 64)
LOCATIONAIP.61154 > LOCATIONBIP.443: Flags [S], cksum 0x271a (correct), seq 697730383, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 552186860 ecr 0,sackOK,eol], length 0
19:03:31.137897 eth0 In IP2 (invalid)
19:03:31.137897 eth0.21 In IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto TCP (6), length 64)
LAPTOPIP.61155 > LOCATIONBIP.443: Flags [S], cksum 0x4c52 (correct), seq 2061410625, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 3113723592 ecr 0,sackOK,eol], length 0
19:03:31.138285 eth1 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [none], proto TCP (6), length 64)
LOCATIONAIP.61155 > LOCATIONBIP.443: Flags [S], cksum 0x2e54 (correct), seq 2061410625, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 3113723592 ecr 0,sackOK,eol], length 0
Nginx gives me no logs whatsoever.
I have Nat reflection. However, turning it off doesn't really help. Neither does it turning off wireguard.
Traffic also works normally if I connect a laptop behind location A to Location B via Wireguard (ie, directly, not via my site-to-site wireguard).
And, don't forget: outside my own net, everything works perfectly.
I fired up a python server on port 8000 on the webserver on location B (and the appropriate port forwarding), and I can't get traffic from this server on location A. However, doing the same on another server on location B - on the same vlan - works. This makes me think that the problem could be on the server, but I don't know.
How can I debug something like this? I'd appreciate any tip! So annoying that this just "happens" without any intervention whatsoever.
It was fail2ban that locked me out. >:(