1. Introduction
One of the attractive features of OPNsense is the ability to run additional services directly on the firewall. HAProxy, Caddy and Nginx offer reverse proxy functionality, TLS termination and centralized certificate management through ACME. Squid provides similar convenience for forward proxy functionality. I have been using this approach myself, and it seems only natural to take advantage of these possibilities instead of maintaining additional systems.
However, while reviewing my HAProxy configuration, I realized that there are some serious security implications which had not been entirely obvious to me before. The most important one is a fundamental conflict between running a proxy on the firewall and using that same firewall to enforce network segmentation.
None of this is specific to HAProxy. The architectural problem applies to Caddy, Nginx, Squid and other proxy services as well, although their exposure to attacks obviously depends on how they are deployed.
2. Why Network Segmentation Matters
Consider a typical infrastructure with separate networks or VLANs for management, publicly accessible services (DMZ), application servers, databases, VMs, containers and internal clients. The firewall controls communication between these networks, allowing only explicitly required connections.
The primary purpose is not just to prevent unauthorized access from the Internet. It is to limit the blast radius of a successful compromise.
In our times of AI-generated attacks, we should assume that sooner or later an Internet-facing application will contain an exploitable vulnerability. If an attacker gains control of a web server, the firewall should prevent that server from becoming a starting point for attacks against the rest of our infrastructure. This is what network segmentation and the principle of least privilege are about.
And this makes sense even in small installations. With Proxmox VE, for example, VMs and LXCs can easily be placed into separate virtual networks, with OPNsense controlling the traffic between them. As long as the segmentation stays within the virtualized environment, not even managed physical switches are required.
3. What Happens When the Proxy Runs on the Firewall?
Consider two architectures:
A: Internet → OPNsense with proxy → Backend services
No question: this is the convenient solution. However, OPNsense permits outgoing connections originating from the firewall itself by default. The automatically generated rule is called "let out anything from firewall host itself". The OPNsense Caddy documentation (https://docs.opnsense.org/manual/how-tos/caddy.html) explicitly confirms that firewall rules allowing the proxy to reach internal services are unnecessary because traffic originating from the firewall is allowed.
Consequently, the firewall has a fundamentally different network position from a server in a DMZ. It generally has routes to all directly connected networks and potentially even others reachable through VPN tunnels. The usual inter-VLAN firewall rules do not automatically restrict connections originating from processes running on the firewall itself.
Squid: No Exploit Required!
With Squid as a forward proxy, the situation can be even worse: No vulnerability is required at all.
In fact, I encountered exactly this problem in my own setup: A client in one VLAN was prohibited by firewall rules from accessing services in another VLAN. However, the same client could reach those services through Squid running on OPNsense!
The reason is simple: Squid establishes the outgoing connections from the firewall itself, effectively bypassing the inter-VLAN restrictions. Unless Squid's own ACLs explicitly prohibit such connections, the network separation is defeated during perfectly normal and intended operation.
This is not a new discovery, either. The problem was already discussed in OPNsense issue #3710 (https://github.com/opnsense/core/issues/3710) back in 2019.
So with Squid, the security boundary can be bypassed without compromising anything.
Now assume an attacker exploits a remote-code-execution vulnerability in the proxy.
The attacker may be able to initiate connections from the firewall host to systems that would have been inaccessible from a separate DMZ host. This effectively eliminates the security boundary we just created. And the attacker does not necessarily need root privileges for this. An ordinary Unix process can initiate outgoing TCP connections. Running the proxy under an unprivileged account or inside a chroot environment does not, by itself, restrict its network access.
B: Internet → OPNsense → Proxy in DMZ → Backend services
Here the proxy resides in its own network, and OPNsense explicitly permits only the connections it needs to reach its backends. If the proxy is compromised, the attacker can potentially attack those authorized backends, but the firewall continues to block access to everything else.
The security boundary remains intact. That was the whole point of network segmentation in the first place.
Of course, locally generated traffic on OPNsense can also be restricted using pf rules. But the firewall itself needs access for DNS, NTP, updates, certificate management, monitoring, VPNs and numerous other purposes. Building and maintaining suitable restrictions is considerably more complicated than applying a simple policy to a separate DMZ host. Furthermore, destination-based rules do not necessarily distinguish legitimate firewall services from a compromised proxy connecting to the same destination.
So why deliberately create separate security zones only to give a potentially exposed application the network privileges of the device enforcing those zones?
4. Side Aspect: Hiding Administrative Services
There is another, somewhat different issue I encountered while reviewing my HAProxy setup.
Sometimes we want to retain remote access to administrative services without exposing them to routine Internet scans.
A good example is Proxmox VE hosting OPNsense as a VM. If OPNsense fails after an update or configuration mistake, we may need SSH access to the PVE host to repair the firewall or start a fallback VM. Obviously, that access must not depend on the failed OPNsense instance. But we also do not necessarily want to advertise our hypervisor's management interface to the entire Internet.
This requirement is not limited to hypervisors. It applies to various management interfaces and other services that should remain available without being easily discoverable.
There are two interesting approaches I often use:
A: A secret IPv6 address
Assign a randomly generated global IPv6 address to the management interface without publishing it in DNS. Even if an attacker knows the /64 prefix, there are 2^64 (approximately 18.4 quintillion) possible interface identifiers.
Provided the address is genuinely unpredictable, exhaustive port scanning is impractical. This is fundamentally different from IPv4, where scanning an entire subnet is usually trivial. Of course, this assumes that the address is not derived from a MAC address, sequential allocation or another predictable pattern. RFC 7707 (https://www.rfc-editor.org/rfc/rfc7707.html) discusses ways of discovering IPv6 hosts without brute-forcing the entire address space.
For the PVE example, the host must also have an Internet connection independent of the OPNsense VM.
B: A secret hostname behind a reverse proxy
Alternatively, we could configure an unpublished virtual hostname such as emergency.example.com on our publicly accessible reverse proxy. Only someone knowing that hostname would supposedly reach the corresponding backend. This also handles the problem that arises from the IPv6-only approach: It can be used for both IPv4 and IPv6.
Unfortunately, this approach has a rather unexpected weakness.
5. TLS SNI Is Not the HTTP Hostname
When establishing an HTTPS connection, the client may provide a hostname through TLS Server Name Indication (SNI), which the server uses to select an appropriate certificate. The actual HTTP hostname, however, is transmitted separately, using the Host header in HTTP/1.1 or :authority in HTTP/2.
These names do not necessarily have to match.
Suppose our proxy serves www.example.com publicly, while pve.example.com and opnsense.example.com are unpublished administrative hostnames.
An attacker establishes a TLS connection using the known SNI name www.example.com. Once connected, the attacker can send requests with different HTTP Host headers, such as pve.example.com or opnsense.example.com.
If the proxy routes requests based on those HTTP hostnames without additional restrictions, it will forward them to the respective backends. A single known SNI name can therefore be sufficient to enumerate otherwise hidden virtual hosts. Persistent HTTP connections can carry multiple such requests without requiring a new TLS handshake for each attempt.
And with typical names like pve, proxmox, opnsense, firewall, nas, backup or grafana, the search space is rather small.
5.1. Another Trap: Accidentally Exposing Internal Services
There is another pitfall when using a reverse proxy simply to provide HTTPS and valid certificates for internal services. If these services share a publicly accessible frontend with other applications, they may inadvertently become accessible from the Internet as well.
Neither internal DNS records nor the absence of public DNS entries prevent this. An attacker can connect to the public proxy address and supply the corresponding hostname. This does not even require an SNI/Host mismatch.
Therefore, internal-only services should use a separate frontend bound to internal interfaces or be protected by explicit access restrictions. And remember: ACME DNS-01 validation allows obtaining publicly trusted certificates without exposing the services themselves.
5.2. Forcing Equality of TLS SNI and Hostname?
Why not simply enforce equality between SNI and Host? This is possible, but not always desirable. HTTP/2 explicitly allows connection reuse across different origins (https://www.rfc-editor.org/rfc/rfc9113.html#section-9.1.1) under certain conditions. Enforcing strict equality can interfere with legitimate clients. And even if enforced, an attacker can still try matching SNI/Host pairs, one connection at a time.
Of course, a randomly generated hostname with sufficient entropy could be just as difficult to guess as a random IPv6 address. The problem is relying on conventional, predictable but unpublished hostnames as a security measure. Also: Who really uses something like afas4gfhrt76gfg.example.com?
6. Don't Accidentally Reveal Your Secrets
Both approaches rely on information remaining unknown. There are several ways this assumption can fail:
DNS enumeration: Depending on zone configuration, hostnames may be discovered through improperly permitted zone transfers, DNSSEC NSEC enumeration, dictionary attacks or historical DNS records. Certificate Transparency logs may also disclose hostnames for which individual public certificates were issued in the past. Wildcard certificates avoid revealing individual subdomains through their certificates, but not through other mechanisms.
For a "secret" IPv6 address, the obvious precaution is to avoid publishing any corresponding DNS records, including AAAA and revealing reverse DNS (PTR) records.
Outgoing IPv6 connections: There is another easily overlooked problem. A host with multiple global IPv6 addresses may select the supposedly secret management address as the source for outgoing connections. If that happens, the remote server learns the address, which may then appear in logs or other records. Its secrecy is gone, even though it was never published in DNS.
A secret IPv6 management address should therefore not be used as the source for ordinary outgoing Internet connections.
Use a separate address for outgoing traffic and verify the actual source address selection. On Linux, "ip -6 route get" can help identify the selected source for a particular destination. Applications explicitly binding to an address must be considered separately. Temporary IPv6 privacy addresses may help, but should not simply be assumed to solve the problem automatically. Hint: Setting the preferred lifetime of the secret IPv6 address to zero marks it as deprecated, making Linux prefer other suitable source addresses for new outgoing connections (RFC 6724, Rule 3).
Finally, neither a secret IP address nor a secret hostname replaces authentication or access control. Both are additional obstacles against discovery, nothing more.
7. Conclusion
Running HAProxy, Caddy, Nginx or Squid directly on OPNsense is extremely convenient. But convenience comes with architectural consequences.
The problem is not necessarily that the proxy software itself is insecure. The problem is what happens when it becomes compromised.
We deliberately segment our networks to contain such incidents. Placing a potentially exposed proxy directly on the firewall may give an attacker a way around exactly those restrictions.
Similarly, hiding administrative services behind unpublished hostnames may offer considerably less protection against discovery than expected, particularly when a publicly accessible reverse proxy provides a known entry point.
Security is not only about preventing a compromise. It is also about limiting the damage when one occurs. Note that I say "when", not "if" here. And the device enforcing those limits is not necessarily the best place to run the applications that could bypass them.
Thus, all of the available proxies for OPNsense should probably carry a large sign: WARNING: LARK'S VOMIT!!! - but then, our sales would plummet! If you don't know what that means, look it up!
Placeholder - maybe I will describe a system that can act as a reverse proxy in a DMZ in the future here.
Caddy can hide SNI inside Encrypted ClientHello (ECH). An attacker could only see the "pooled" name used for ECH then. So you can make the actual name "secret" with that.
This is information regardless of where you run the proxy, running it in a DMZ is a good choice for separation of concerns, just wanted to add this tidbit
Thanks, Cedrik, good point! ECH certainly helps against passive observation of SNI.
However, my concern is active enumeration: An attacker connects to the public proxy using a known SNI name (e.g. www.example.com), then tries different HTTP Host headers (pve.example.com, opnsense.example.com, etc.) over that same TLS connection to discover hidden backends.
As far as I understand, ECH doesn't prevent this, since the attacker is a client, not a passive observer. Or am I missing something?
No, I don't think it would protect against bruteforcing.
A DNS label could be very long and cryptic as well though... and without being able to observe it passively via ECH it could be impossible to guess.
But that's all mostly security through obscurity...
The only "real" way is to have an air gapped environment, then the AI cannot crawl you :)
security is the direct opposite of convenience xD
I prefer Crunchy Frog. 😬
Thanks for the great writeup.
Yea ECH will only "hide it" from a tranzit perspetive.
Very nice write up, informative and who doesn't like good security based blog posts?
I run RPs behind the out of the firewall, separate instances in a Cluster (Keepalived for the win) and one of the reasons was exactly as you mentioned in your write up;
"let out anything from firewall host itself" + "maintaining suitable restrictions is considerably more complicated".
Regards,
S.
Quote from: meyergru on Today at 11:24:38 AM4. Side Aspect: Hiding Administrative Services
There is another, somewhat different issue I encountered while reviewing my HAProxy setup.
Sometimes we want to retain remote access to administrative services without exposing them to routine Internet scans.
A good example is Proxmox VE hosting OPNsense as a VM. If OPNsense fails after an update or configuration mistake, we may need SSH access to the PVE host to repair the firewall or start a fallback VM. Obviously, that access must not depend on the failed OPNsense instance. But we also do not necessarily want to advertise our hypervisor's management interface to the entire Internet.
This requirement is not limited to hypervisors. It applies to various management interfaces and other services that should remain available without being easily discoverable.
That's what OOB devices are for ?!
Another option is to login from a completely different system where you can create a temporary NAT session for the specific IPMI/iDRAC/iLO or Server IP Address and use that to fix whatever needs fixing.
If you want you can combine that with let's say Wireguard VPN connectivity :)
Anyway...
I am 100% a fan of NOT running any kind of Reverse Proxy on a Firewall/Router so I hope a lot of people will read this topic in the future!
Some other common stuff I would also keep away from my Firewall/Router :
- DNS because I use Pi-Hole + Unbound on a seperate device.
- VPN because Certificates are annoying + This Reverse Proxy write up reasoning.
- Multicast DNS Bridging that some people do between their regular VLAN and IoT VLAN or for IPTV purposes, because rebooting your Firewall/Router because IGMP Proxy stopped working properly is
NOT COOL !!!Also the reason why I point people to something like this : https://berthub.eu/articles/posts/kpn-interactieve-tv-zelf-doen/
IMHO the best solution for IPTV users :)