Messages

Thanks, Cedrik, good point! ECH certainly helps against passive observation of SNI.

However, my concern is active enumeration: An attacker connects to the public proxy using a known SNI name (e.g. www.example.com), then tries different HTTP Host headers (pve.example.com, opnsense.example.com, etc.) over that same TLS connection to discover hidden backends.

As far as I understand, ECH doesn't prevent this, since the attacker is a client, not a passive observer. Or am I missing something?

Placeholder - maybe I will describe a system that can act as a reverse proxy in a DMZ in the future here.

[1. Introduction]

1. Introduction

One of the attractive features of OPNsense is the ability to run additional services directly on the firewall. HAProxy, Caddy and Nginx offer reverse proxy functionality, TLS termination and centralized certificate management through ACME. Squid provides similar convenience for forward proxy functionality. I have been using this approach myself, and it seems only natural to take advantage of these possibilities instead of maintaining additional systems.

However, while reviewing my HAProxy configuration, I realized that there are some serious security implications which had not been entirely obvious to me before. The most important one is a fundamental conflict between running a proxy on the firewall and using that same firewall to enforce network segmentation.

None of this is specific to HAProxy. The architectural problem applies to Caddy, Nginx, Squid and other proxy services as well, although their exposure to attacks obviously depends on how they are deployed.

2. Why Network Segmentation Matters

Consider a typical infrastructure with separate networks or VLANs for management, publicly accessible services (DMZ), application servers, databases, VMs, containers and internal clients. The firewall controls communication between these networks, allowing only explicitly required connections.

The primary purpose is not just to prevent unauthorized access from the Internet. It is to limit the blast radius of a successful compromise.

In our times of AI-generated attacks, we should assume that sooner or later an Internet-facing application will contain an exploitable vulnerability. If an attacker gains control of a web server, the firewall should prevent that server from becoming a starting point for attacks against the rest of our infrastructure. This is what network segmentation and the principle of least privilege are about.

And this makes sense even in small installations. With Proxmox VE, for example, VMs and LXCs can easily be placed into separate virtual networks, with OPNsense controlling the traffic between them. As long as the segmentation stays within the virtualized environment, not even managed physical switches are required.

3. What Happens When the Proxy Runs on the Firewall?

Consider two architectures:

A: Internet → OPNsense with proxy → Backend services

No question: this is the convenient solution. However, OPNsense permits outgoing connections originating from the firewall itself by default. The automatically generated rule is called "let out anything from firewall host itself". The OPNsense Caddy documentation explicitly confirms that firewall rules allowing the proxy to reach internal services are unnecessary because traffic originating from the firewall is allowed.

Consequently, the firewall has a fundamentally different network position from a server in a DMZ. It generally has routes to all directly connected networks and potentially even others reachable through VPN tunnels. The usual inter-VLAN firewall rules do not automatically restrict connections originating from processes running on the firewall itself.

Squid: No Exploit Required!

With Squid as a forward proxy, the situation can be even worse: No vulnerability is required at all.

In fact, I encountered exactly this problem in my own setup: A client in one VLAN was prohibited by firewall rules from accessing services in another VLAN. However, the same client could reach those services through Squid running on OPNsense!

The reason is simple: Squid establishes the outgoing connections from the firewall itself, effectively bypassing the inter-VLAN restrictions. Unless Squid's own ACLs explicitly prohibit such connections, the network separation is defeated during perfectly normal and intended operation.

This is not a new discovery, either. The problem was already discussed in OPNsense issue #3710 back in 2019.

So with Squid, the security boundary can be bypassed without compromising anything.

Now assume an attacker exploits a remote-code-execution vulnerability in the proxy.

The attacker may be able to initiate connections from the firewall host to systems that would have been inaccessible from a separate DMZ host. This effectively eliminates the security boundary we just created. And the attacker does not necessarily need root privileges for this. An ordinary Unix process can initiate outgoing TCP connections. Running the proxy under an unprivileged account or inside a chroot environment does not, by itself, restrict its network access.

B: Internet → OPNsense → Proxy in DMZ → Backend services

Here the proxy resides in its own network, and OPNsense explicitly permits only the connections it needs to reach its backends. If the proxy is compromised, the attacker can potentially attack those authorized backends, but the firewall continues to block access to everything else.

The security boundary remains intact. That was the whole point of network segmentation in the first place.

Of course, locally generated traffic on OPNsense can also be restricted using pf rules. But the firewall itself needs access for DNS, NTP, updates, certificate management, monitoring, VPNs and numerous other purposes. Building and maintaining suitable restrictions is considerably more complicated than applying a simple policy to a separate DMZ host. Furthermore, destination-based rules do not necessarily distinguish legitimate firewall services from a compromised proxy connecting to the same destination.

So why deliberately create separate security zones only to give a potentially exposed application the network privileges of the device enforcing those zones?

4. Side Aspect: Hiding Administrative Services

There is another, somewhat different issue I encountered while reviewing my HAProxy setup.

Sometimes we want to retain remote access to administrative services without exposing them to routine Internet scans.

A good example is Proxmox VE hosting OPNsense as a VM. If OPNsense fails after an update or configuration mistake, we may need SSH access to the PVE host to repair the firewall or start a fallback VM. Obviously, that access must not depend on the failed OPNsense instance. But we also do not necessarily want to advertise our hypervisor's management interface to the entire Internet.

This requirement is not limited to hypervisors. It applies to various management interfaces and other services that should remain available without being easily discoverable.

There are two interesting approaches I often use:

A: A secret IPv6 address

Assign a randomly generated global IPv6 address to the management interface without publishing it in DNS. Even if an attacker knows the /64 prefix, there are 2^64 (approximately 18.4 quintillion) possible interface identifiers.

Provided the address is genuinely unpredictable, exhaustive port scanning is impractical. This is fundamentally different from IPv4, where scanning an entire subnet is usually trivial. Of course, this assumes that the address is not derived from a MAC address, sequential allocation or another predictable pattern. RFC 7707 discusses ways of discovering IPv6 hosts without brute-forcing the entire address space.

For the PVE example, the host must also have an Internet connection independent of the OPNsense VM.

B: A secret hostname behind a reverse proxy

Alternatively, we could configure an unpublished virtual hostname such as emergency.example.com on our publicly accessible reverse proxy. Only someone knowing that hostname would supposedly reach the corresponding backend. This also handles the problem that arises from the IPv6-only approach: It can be used for both IPv4 and IPv6.

Unfortunately, this approach has a rather unexpected weakness.

5. TLS SNI Is Not the HTTP Hostname

When establishing an HTTPS connection, the client may provide a hostname through TLS Server Name Indication (SNI), which the server uses to select an appropriate certificate. The actual HTTP hostname, however, is transmitted separately, using the Host header in HTTP/1.1 or :authority in HTTP/2.

These names do not necessarily have to match.

Suppose our proxy serves www.example.com publicly, while pve.example.com and opnsense.example.com are unpublished administrative hostnames.

An attacker establishes a TLS connection using the known SNI name www.example.com. Once connected, the attacker can send requests with different HTTP Host headers, such as pve.example.com or opnsense.example.com.

If the proxy routes requests based on those HTTP hostnames without additional restrictions, it will forward them to the respective backends. A single known SNI name can therefore be sufficient to enumerate otherwise hidden virtual hosts. Persistent HTTP connections can carry multiple such requests without requiring a new TLS handshake for each attempt.

And with typical names like pve, proxmox, opnsense, firewall, nas, backup or grafana, the search space is rather small.

5.1. Another Trap: Accidentally Exposing Internal Services

There is another pitfall when using a reverse proxy simply to provide HTTPS and valid certificates for internal services. If these services share a publicly accessible frontend with other applications, they may inadvertently become accessible from the Internet as well.

Neither internal DNS records nor the absence of public DNS entries prevent this. An attacker can connect to the public proxy address and supply the corresponding hostname. This does not even require an SNI/Host mismatch.

Therefore, internal-only services should use a separate frontend bound to internal interfaces or be protected by explicit access restrictions. And remember: ACME DNS-01 validation allows obtaining publicly trusted certificates without exposing the services themselves.

5.2. Forcing Equality of TLS SNI and Hostname?

Why not simply enforce equality between SNI and Host? This is possible, but not always desirable. HTTP/2 explicitly allows connection reuse across different origins under certain conditions. Enforcing strict equality can interfere with legitimate clients. And even if enforced, an attacker can still try matching SNI/Host pairs, one connection at a time.

Of course, a randomly generated hostname with sufficient entropy could be just as difficult to guess as a random IPv6 address. The problem is relying on conventional, predictable but unpublished hostnames as a security measure. Also: Who really uses something like afas4gfhrt76gfg.example.com?

6. Don't Accidentally Reveal Your Secrets

Both approaches rely on information remaining unknown. There are several ways this assumption can fail:

DNS enumeration: Depending on zone configuration, hostnames may be discovered through improperly permitted zone transfers, DNSSEC NSEC enumeration, dictionary attacks or historical DNS records. Certificate Transparency logs may also disclose hostnames for which individual public certificates were issued in the past. Wildcard certificates avoid revealing individual subdomains through their certificates, but not through other mechanisms.

For a "secret" IPv6 address, the obvious precaution is to avoid publishing any corresponding DNS records, including AAAA and revealing reverse DNS (PTR) records.

Outgoing IPv6 connections: There is another easily overlooked problem. A host with multiple global IPv6 addresses may select the supposedly secret management address as the source for outgoing connections. If that happens, the remote server learns the address, which may then appear in logs or other records. Its secrecy is gone, even though it was never published in DNS.

A secret IPv6 management address should therefore not be used as the source for ordinary outgoing Internet connections.

Use a separate address for outgoing traffic and verify the actual source address selection. On Linux, "ip -6 route get" can help identify the selected source for a particular destination. Applications explicitly binding to an address must be considered separately. Temporary IPv6 privacy addresses may help, but should not simply be assumed to solve the problem automatically. Hint: Setting the preferred lifetime of the secret IPv6 address to zero marks it as deprecated, making Linux prefer other suitable source addresses for new outgoing connections (RFC 6724, Rule 3).

Finally, neither a secret IP address nor a secret hostname replaces authentication or access control. Both are additional obstacles against discovery, nothing more.

7. Conclusion

Running HAProxy, Caddy, Nginx or Squid directly on OPNsense is extremely convenient. But convenience comes with architectural consequences.

The problem is not necessarily that the proxy software itself is insecure. The problem is what happens when it becomes compromised.

We deliberately segment our networks to contain such incidents. Placing a potentially exposed proxy directly on the firewall may give an attacker a way around exactly those restrictions.

Similarly, hiding administrative services behind unpublished hostnames may offer considerably less protection against discovery than expected, particularly when a publicly accessible reverse proxy provides a known entry point.

Security is not only about preventing a compromise. It is also about limiting the damage when one occurs. Note that I say "when", not "if" here. And the device enforcing those limits is not necessarily the best place to run the applications that could bypass them.

Thus, all of the available proxies for OPNsense should probably carry a large sign: WARNING: LARK'S VOMIT!!! - but then, our sales would plummet! If you don't know what that means, look it up!

[und]

Gut, ich nehme an, das ist das betroffene Interface-Netz.

Interessant wäre der Vergleich zu den Firewall-Regeln der funktionierenden Interfaces, aber eins sieht man schon jetzt:
Deine "Default allow LAN to any rule" erlaubt bei Dir nur spezifischen TCP-Traffic. Du hast offenbar die Default-Regel für das erste LAN angepasst, ohne die Bezeichnung zu ändern.

Am Rande bemerkt: Je nachdem, zu welchem Zweck man VLANs anlegt, schottet man diese normalerweise gegeneinander ab. Falls die Segmentierung ausschließlich der Trennung von Broadcast Domains zur Vermeidung von zu viel Broadcast Traffic dient, gilt das natürlich nicht.
Will sagen: Die normale "Default allow LAN to any rule" ist sinngemäß sinnvoll für kleine Heimnetze ohne VLANs, mit VLANs würde man zuvorderst mal die anderen (V)LANs schützen und trotzdem (bestimmten) Internet-Verkehr zulassen - es gibt einige Threads dazu, u.a. wie man limitierte Netze zu Gruppen zusammenfasst, um das zentral zu regeln.

Bei Deiner Konfiguration geht mehr (nämlich Zugriff auf andere VLANs)  und weniger (z.B. kein Ping) als das, also würde ich annehmen, dass sie noch weitere Fragen aufwirft. Beispielsweise ist nicht nur Ping betroffen, wenn Du explizit nur bestimmte TCP-Ports erlaubst: Wie soll dann z.B. DNS funktionieren und andere Firewall-Services?

Ich will nichts unterstellen, aber vielleicht liest Du zuerst mal dies - oder sogar dies, je nachdem, was Du vorhast.

If only Mikrotik configuration was not such a hassle... I like Unifi for everything switching and APs.

Having done 802.1x for ~200.000 clients in an enterprise environment, I think the real benefits are unachievable in a private environment:

You need RADIUS-capable switching equipment for that (implemented correctly, unlike with some Unifi switches in the recent past) and also, MAC-based 802.1x can certainly be circumvented. Some client devices do not even work with that.

On the other hand, "secure" certificate-based 802.1x is hard to handle and you need client OS support, which many IoT devices essentially lack.

The situation is a little better for WiFi, since it has inherent access limiting capabilities.

But @Patrick: device isolation is useless in many contexts, IMHO: think of media sharing and broadcast-based traffic. I just had a Miele hob not contacting its extractor hood because of client isolation. Also, it does not even work over different APs, only for devices connected to the same AP.

How about the official documentation for a start?

[multiple gateways and policy-based routing (PBR)]

Your latest tcpdump provides an important clue: Even in the broken state, DNS queries from 10.127.127.2 reach 10.127.127.1, and valid DNS responses are sent back through wg4.

This suggests that Unbound is working. The problem is more likely related to packet delivery, firewall rules, or routing/gateway selection during startup.

Since your setup involves multiple gateways and policy-based routing (PBR), I particularly suspect a routing or firewall state issue during boot, possibly involving incorrect gateway selection or asymmetric routing.

Could you perform the following tests before restarting WireGuard, while simultaneously running `tcpdump -ni wg4 'host 10.127.127.2'` on OPNsense?

1. Explicit DNS lookup: Query `10.127.127.1` directly from the affected client (e.g. `nslookup example.com 10.127.127.1`). If this fails despite DNS responses appearing in the capture, the problem is likely on the return path to the client or in the client's DNS stack. If it succeeds, DNS transport itself is working.

2. Direct IP connection: Try accessing a known reachable Internet server by IP address, bypassing DNS entirely. If this fails while DNS works, suspect forwarding, NAT, firewall rules, or gateway selection. The capture will show whether the connection attempt reaches wg4.

3. Connection by hostname: Access the same server by hostname. If direct IP access works but hostname access fails, investigate the client's DNS configuration or resolver behavior rather than general IP routing.

Given your multiple gateways and PBR rules, I would also compare the firewall states and gateway selection for the affected client traffic before and after restarting WireGuard.

These tests should distinguish DNS transport issues from forwarding/routing problems and incorrect client-side name resolution.

I would avoid introducing startup delays until we have identified the failing component.

Generally, there are usually one of these problems (not all with all DHCP servers, but hey):

1. With some servers (at least ISC), the creation of a static reservation does not get active unless the service is restarted.
2. Some servers (also ISC), you should not have static reservations within dynamic ranges.
3. When you let a client get a dynamic reservation and convert it to a static one, the dynamic reservation is not deleted, such that a reboot still gives the old IP. You have to manually clean the dynamic reservation from the lease file (and restart the service to recognize that fact).
4. With newer services, a DUID ist being used instead of a MAC (unless you change that setting) - which may give the impression that static reservations (for the MAC) do not work.

That being said, I know that is not an exact answer, just because I happen to use Kea and not DNSmasq.

NVME SSDs unterscheiden sich ziemlich in den Controllern. Bei Geizhals kannst Du sehen, wieviel die Modelle im Betrieb brauchen. Es gibt welche, die bei >7 Watt liegen, das ist mehr als ein N100. Bei der minimalen thermischen Trägheit und winzigen Abstrahlfläche müssen die also heiß werden, zudem die Hitze sich nicht gleichmäßig auf die Fläche verteilt.

Es gibt passive Kühlkörper, die man aufsetzen kann, die oft >10°C Abkühlung bringen.

Ich suche gute NVME-SSDs für ZFS danach aus, ob sie geringe Leistung brauchen, hohen TBW-Wert und möglichst DRAM-Cache haben. Solche Modelle sind dann leider auch etwas teurer.

Eine weitere Möglichkeit ist, die SSD mittel eines Kommandos in einen geringeren Power-State zu bringen - sie agiert dann langsamer, aber sparsamer. Man sieht die möglichen States mit "nvmecontrol power -l nvme0".

Your time might be better spent by creating a PR for this: https://github.com/opnsense/core/issues/8239

While I generally like the idea, I personally refrain from applying non-vetted patches or packages to a security-relevant appliance like OPNsense.

Aha. Found it:

https://github.com/flaviuvlaicu/opnsense-topo-map

I had a look at the code. The plugin does not actually discover the network topology. It merely collects clients from ARP/Kea; the relationships between clients, switches and APs are then defined manually by the user via drag & drop and stored in topology.json.

There is no LLDP, SNMP, switch MAC-table or controller data being queried. Therefore, the plugin cannot determine which switch/port a client is actually connected through, nor can it detect when a wireless client roams from one AP to another.

So essentially, it is a graphical network diagram editor with an automatically populated client list, not an automatic topology discovery tool.

That said, I would be rather cautious about installing software from arbitrary GitHub repositories on an OPNsense firewall.

So do I.

OpnSense does not know anything about this other than on which interface a client is connected to, so what you want is not feasible using it.

There is network equipment available that can show you things like that, however. Take a look at Unifi switches and APs. My personal opinion is that they are quite good at making those, less good at making routers (I use OpnSense for that).

You do not need a dream box or any of their routers, though. The network controller can be run on x64 hardware or as a VM (the software is called Unifi OS server and is free).

That is a collision of interest of the purest kind.

You can block DoT/DoH, but obviously, you cannot intercept and modify that traffic, because it is encrypted and protected via certificates. Normally, such an approach would be used in conjunction with a traffic interception of normal, unencrypted DNS traffic to be able to block certain DNS domains. If anyone wants to use your network, he must fall back to standard DNS in such a scenario, i.e. abide by your imposed rules.

If you want to give a device internet access that does not play by those rules, you can, but then you cannot block DoT/DoH, as you have seen.
In that case, I would put that device into a separate VLAN (probably also a separate WLAN), such that it can do whatever it chooses, but has no possibility to access any of your own LAN ressources. Of course, you can allow specific services, like accessing a printer.

Or to put it short: You cannot have the cake and eat it, too.

BTW: If there was a canary domain, you would be able to see a DNS request (and, obviously, it would be via normal DNS).

Since Chromium deliberately chose not to implement a Firefox-style canary domain, there is no DNS-side mechanism to force such a fallback; and if the Chromebook is managed by the school, the DoH-only setting is most likely policy-locked anyway.