Trouble accessing HTTP-only web sites via port 80

[tonys]

OPNsense 24.1.8-amd64
FreeBSD 13.2-RELEASE-p11
OpenSSL 3.0.13
Suricata 7.0.5_1

I have an ongoing problem wherein LAN access to certain HTTP web sites are being blocked. Here are a few sites I'm unable to access:

1. http://repo.feed.flightradar24.com (using "sudo apt update" under RaspberryPi OS)
    Err:7 http://repo.feed.flightradar24.com flightradar24 InRelease Connection failed [IP: 52.217.205.24 80] 

2. Renewing LetsEncyrpt SSL certificate: (using "sudo certbot renew" under Ubuntu Linux 22.04)
   The Certificate Authority failed to download the challenge files from the temporary standalone webserver started    by Certbot on port 80     

3. Also, a few Facebook pages that don't use HTTPS are being blocked (e.g. via iOS access on iPhones)

Considering these are outbound port 80 requests to external sites, I assume some firewall rule must be blocking them on the way in from the various LAN servers. I can't be sure though because it's really not clear where the blockages are occurring. Digging through the log and json files in /var/log/filter and /var/log/suricata, I found references to the IP address in issue 1 above but don't understand what they're telling me:

/var/log/filter/latest.log:

<134>1 2024-06-01T15:40:47-05:00 OPNsense.home.lan filterlog 4446 - [meta sequenceId="114698"] 10,,,0,igc1,match,nat,out,4,0x10,,64,0,0,DF,6,tcp,40,192.168.1.10,52.217.205.24,43298,80,0,S,64240,,0,,
filter_20240601.log:<134>1 2024-06-01T15:40:47-05:00 OPNsense.home.lan filterlog 4446 - [meta sequenceId="114699"] 101,,,e3758d9e17f4ad487875821fc183e910,igc1,match,pass,out,4,0x10,,64,0,0,DF,6,tcp,40,67.163.46.185,52.217.205.24,64548,80,0,S,64240,,0,,     

/var/log/suricata/eve.json:

{"timestamp":"2024-06-01T15:41:10.070699-0500","flow_id":1725326563306316,"in_iface":"igc0","event_type":"alert","src_ip":"192.168.1.10","src_port":43298,"dest_ip":"52.217.205.24","dest_port":80,"proto":"TCP","pkt_src":"wire/pcap","tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":2013504,"rev":6,"signature":"ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management","category":"Not Suspicious Traffic","severity":3,"metadata":{"created_at":["2011_08_31"],"updated_at":["2020_04_22"]}},"http":{"hostname":"repo.feed.flightradar24.com","url":"/dists/flightradar24/InRelease","http_user_agent":"Debian APT-HTTP/1.3 (2.2.4)","http_method":"GET","protocol":"HTTP/1.1","length":0},"app_proto":"http","direction":"to_server","flow":{"pkts_toserver":10,"pkts_toclient":3,"bytes_toserver":822,"bytes_toclient":166,"start":"2024-06-01T15:40:46.991532-0500","src_ip":"192.168.1.10","dest_ip":"52.217.205.24","src_port":43298,"dest_port":80}}
                   
How do I track these down? I've been all over the OPNSense pages for weeks but I'm not making any progress. Any help would be greatly appreciated.

[cookiemonster]

those seem to be just the record of the connections going out. Default firewall rules won't stop traffic out to 80 or any port. Default LAN is allow all out.
Maybe something on the clients.

[Monviech (Cedrik)]

Certbot renewal is an inbound port 80 connection, it needs a port forward and a firewall rule to allow it, or for ipv6 just a firewall rule to allow it.

If you have multiple webservers behind the OPNsense, consider using a reverse proxy. In for example, os-caddy, you can reverse proxy the ACME HTTP-01 challenge to several servers behind the OPNsense easily, while using it as reverse proxy at the same time.

https://docs.opnsense.org/manual/how-tos/caddy.html#redirect-acme-http-01-challenge

[tonys]

@Monviech,

The attached screen shots show the following:

1. OPnsense port forwards to my DMZ server on 192.168.30.3 (80, 443, 943, 1194)
2. OPNsense firewall rules to the the DMZ server
3. UFW firewall rules on the DMZ server itself
4. LetsEncrypt failure when updating over port 80

I have an OpenVPN webserver behind OPNsense which works fine on ports 443, 943, and 1194. LetsEncrypt requires port 80 to be open during cert updates which is also forwarded, firewall-ruled and open on the OpenVPN server. OpenVPN itself works fine in this configuration and all incoming requests on ports 443, 943, and 1194 work fine. However, LetsEncrypt over port 80 doesn't work. I'm baffled by this as I have the same settings for port 80 as for the other ports.

[Monviech (Cedrik)]

Did you check System - Settings - Administration that the OPNsense WebGUI runs on a different TCP port than 443 and that the web gui redirect rule is disabled (to free port 80)?

[tonys]

Attached are my current System settings. The web GUI redirect is indeed disabled. However, I'm using 443 for OPNsense which does NOT impact OpenVPN access from an external IP address. Users outside my home network can find and log into OpenVPN.

The port 80 replies from LetsEncrypt are MIA. Can you please explain why I would need to change the OPNsense GUI to a different port (e.g. 1443?) to fix port 80 replies? I find this very confusing.

UPDATE: I changed the port to 1443 but this did NOT fix the LetsEcrypt renewal. I get the same error so I'm setting the HTTPS port back to 443 for the time being.

Thanks,
Tony

[Monviech (Cedrik)]

Well it does no impact openvpn cause the webgui uses tcp port 443 and openvpn uses udp port 443. So thats okay.

I thought you had a webserver listening on tcp port 80 and 443 behind the OPNsense thats why I suggested that.

EDIT:

Since you have IPv6, why not allow access on port 80 to the SLAAC address of your ACME Client? It could try to solve the challenge on the IPv6 address but nothing is there since in your DNS Record its the address of the OPNsense (probably).

It would really help if you would use a reverse proxy there, it would make sure the challenge gets reverse proxied to the right backend endpoint while using your frontend IPv4 and IPv6 addresses.

[tonys]

"Well it does no impact openvpn cause the webgui uses tcp port 443 and openvpn uses udp port 443. So thats okay."

Clarification of OpenVPN ports:

TCP 443: used only for client web access from external IP addresses. Never used unless the OpenVPN client fails.
TCP 943: used for admin web access from external IP addresses. This is used to maintain OpenVPN.
UDP 1194: used for all traffic to/from the OpenVPN server with external clients. UDP 1194 is the normal operating mode for OpenVPN clients. TCP port 443 can be used for better reliability but slower traffic processing. There is no UDP 443 that I've found in the OpenVPN documentation.

What confuses me is that there are identical port forwards and firewall rules for ports 80, 443, 943, and 1194 but only port 80 fails. I don't understand why port 80 is so "special" to OPNsense that it doesn't follow the same rules as 443, 943, and 1194.

Also confusing is that I can access HTTP ports on port 80 from any device's web browser and replies come back to the correct device as expected. The only failing reply is LetsEncrypt which starts a web server on port 80 of the OpenVPN server during certbot renewal. I assume LetsEncrypt sends a request over port 80 but I'm unable to see this traffic moving through the firewall in either direction. There's no documentation of how the web server LetsEncrypt running on port 80 communicates to the outside world and I have yet to be able to see this traffic.

As an aside, I tried using lego to generate certificates via port 443 on the OpenVPN server and this almost works. By almost, I mean the communication process with the outside world via port 443 is successful but the certs I get back don't work with OpenVPN for some reason.

I'll investigate your suggestion regarding SLACC on the acme client but I don't understand anything about this process. I'll also investigate the reverse proxy but again, I have no understanding of this process either.
 

[Monviech (Cedrik)]

Lets assume your opnsense has these IP addresses:
WAN:
1.1.1.1/32
2003::1/64

You have created an A and AAAA record for example.com

example.com in A 1.1.1.1
example.com in AAAA 2003::1

And we assume you have a client with ACME client on port 80 with the following IPs:

192.168.1.1/24
2003::3:1/64

Since there is IPv6, the ACME Lets Encrypt server will try to connect to example.com:80 which will resolve to 2003::1 - which is the OPNsense.

Your client 2003::3:1 will not receive that paket, since it is not the target.

The AAAA record of example.com has to point to the IPv6 address of the internal client, and the Firewall has to allow that.

These are just assumptions right now though based on the provided info and nslookup your domain.