Messages

You did neither tell your configuration nor the topology, so this is only guesswork:

1. Mixing up tagged and untagged (V)LANs on FreeBSD/OpnSense does not work too well. You should have either one or the other.
2. Some hardware does mix up VLANs as well, especially cheap switches. For example, if you cascade switches, an edge switch might short-circuit both VLANs, which can lead to all kinds of problems.

bei den MTUs hab ich nachgeschaut, da steht in der schnittstellenbeschreibung der Anbieter, 1492 drin
so ist es auch auf im Interface auf der OpnSense, als "calculated"

Wie oft heißt es hier, ja - aber. 1492 ist Default, manche ISPs lassen Mini-Jumbo-Frames zu.

930 MBit/s ist angesichts des PPPoE-Overheads so ziemlich das Maximum für Gigabit - deshalb sind ja ONTs mit 2.5 Gigabit manchmal hilfreich, insbsondere, wenn der ISP überprovisioniert, wie meiner.

Einbrüche können diverse Ursachen haben, manchmal sind es schlechte ONTs (oder hier: Medienkonverter), manchmal auch einfach Überprovisionierung seitens des ISP. GPON kann 2.5 GBps Downstream, de facto ist das aber meistens ein "Shared Medium" - deshalb können andere Nutzer auf dem selben OLT-Port Deinen Speed verringern.

Da stellt sich doch zuerst die Frage, ob Du die gebuchten Werte erreichst oder nicht?

Ansonsten ist das ein weites Feld, bei dem viel schiefgehen kann, u.a.:

- schlechte/zu kleine Buffer in manchen ONTs
- Geschwindigkeit bei PPPoE liegt normalerweise unterhalb 1 GBit, falls Link Router <-> ONT auf Gigabit limitiert ist (bei SFP ist eine Umgehung schwierig, weil SGMII oft nicht unterstützt wird, bei Ethernet kann man 2.5Gbos nur verwenden, wenn NIC und ONT das unterstützen).
- Mini-Jumbo-Frames (falls unterstützt) können die MTU wieder auf 1500 Bytes heben.

No, because if you read the documentation closely, you'll see that the design deliberately abstracts from the specific network topology. There isn't necessarily a dedicated WAN port on the Proxmox host to monitor. In my case, for example, the ONT is connected to a switch port.

More importantly, a link-up event does not tell you whether the physical firewall has been shut down. Accidentally connecting something to that port could activate the emergency instance while the physical firewall is still running, resulting in conflicting IP and MAC addresses. Besides, monitoring ports on PVE is far more intrusive than what I would want security-wise.

The intention is to keep the cold spare in maintenance mode during normal operation, so it can be updated regularly and independently of the physical firewall. Switching to emergency mode is a deliberate manual operation after ensuring that the physical firewall is no longer active.

This is a **cold spare**, not an automatic failover solution. If you want automatic failover, use a proper HA setup.

[virtual cold spare]

I used to keep a second, physical OPNsense as a cold spare, just in case I messed up the primary firewall or its hardware failed. There are two annoyances with that approach: keeping the spare updated, and getting the current production configuration onto it precisely when the primary box may no longer be available.

Also, hardware has become expensive. A small China box suitable for the job used to cost around €300; similar hardware can now easily be closer to €600, depending on the model.

So I thought: why not use a virtual cold spare on an existing Proxmox host instead?

The idea is straightforward. The spare OPNsense normally runs as an ordinary maintenance client behind the real firewall, with its own IP and MAC address. You can boot it, update it and test it without taking down your network. If the primary firewall fails, shut it down, switch the VM to emergency mode, and optionally import a current physical OPNsense config.xml. The VM then uses the original interface names, MAC addresses and network connections.

This is not HA, CARP, pfsync or automatic failover. It is a manually activated cold spare, designed to be comparatively simple and cheap.

Source code and configuration examples: https://github.com/meyergru/opnsense-spare

Current status: Automated tests pass, but the generalized, single-hook implementation has not yet completed a full Proxmox/OPNsense boot, WAN cutover and failback integration test. Please test in an isolated environment before relying on it as a recovery procedure.

Prerequisites

• A Proxmox VE host with sufficient spare CPU/RAM/storage to run an OPNsense VM, ideally on a machine independent of the physical firewall.
• A physical OPNsense and a reasonably recent backup of its config.xml, accessible even if the firewall is completely dead.
• A Layer-2 path from each physical firewall network to the corresponding virtual NIC. This may use dedicated PVE NICs/bridges, VLANs on an existing trunk, or a combination. Neither UniFi nor VLANs are mandatory.
• The physical firewall's interface names and MAC addresses, recorded in advance. Access to the Proxmox console is essential while setting this up.
• An OPNsense guest installed with VirtIO NICs (vtnet0, vtnet1, etc.); Python 3 and xorriso on Proxmox.

Step by step

1. Install OPNsense in a VM

For general guidance on installing and tuning OPNsense under Proxmox, see my earlier OPNsense under virtualisation (Proxmox et al.). The instructions below concentrate on the additional requirements for the cold-spare setup.

Create a normal OPNsense VM on Proxmox. For the firmware, select OVMF (UEFI) and add an EFI Disk (VM -> Options -> BIOS, and VM -> Hardware -> Add -> EFI Disk). The physical OPNsense may boot via legacy BIOS; that does not matter because we import its configuration, not its boot disk. The firmware type is not part of config.xml.

Give the VM as many VirtIO network adapters as the physical firewall requires. Keep their net0, net1, etc. ordering stable. Disable automatic VM startup. Set up the necessary Linux bridges first.

Add a serial console from the outset, even when using the graphical installer and VGA as the normal console. On the Proxmox host, with the VM stopped:

Code Select Expand

qm set <VMID> --serial0 socket

Alternatively, use VM -> Hardware -> Add -> Serial Port, select port 0 and socket. Leave a VGA display available for the normal noVNC console. On the host, connect to the serial port with:

Code Select Expand

qm terminal <VMID>

If you prefer a serial console in the browser, change VM -> Hardware -> Display to Serial terminal 0, then open the VM's Console tab (xterm.js). This selects serial output instead of the normal VGA/noVNC view. For actual serial output from OPNsense, go to System -> Settings -> Administration -> Console and enable Serial console as the primary or secondary console (usually 115200 baud). The VM port by itself does not redirect the guest's output. Test both console paths, including after an emergency configuration import: the imported settings may change the primary console.

Install OPNsense normally and initially use it as a maintenance LAN client: separate IP, separate MAC, default gateway via your existing physical firewall. It must not start competing DHCP, router advertisements or other production gateway services on that network. You should be able to reach its GUI from the LAN.

Why are interface names important? Your physical firewall's config.xml may refer to ix1, igc3, etc. In a VirtIO VM the devices are normally called vtnet0, vtnet1. Restoring the configuration without addressing this difference will not reproduce the physical interface assignments.

Rather than editing the imported XML, the spare changes the FreeBSD interface names very early in the guest's boot process, before OPNsense configures its network:

Code Select Expand

Proxmox net0 -> vtnet0 -> renamed to ix1  -> config.xml uses ix1
Proxmox net1 -> vtnet1 -> renamed to igc3 -> config.xml uses igc3

This changes the name, not the underlying VirtIO driver. Hardware-specific tunables, offloading or IPS configurations can still require adjustment.

Why must Maintenance and Emergency use different MAC addresses?

The same VM plays two very different roles:

• Maintenance means coexistence. The physical OPNsense remains online while the VM runs as an ordinary LAN client. Therefore the VM needs its own LAN IP address AND its own MAC address. A different IP alone is not enough: two active NICs with the same MAC on the same Layer-2 network cause ambiguous switch forwarding (MAC-table flapping), even if their IP addresses differ. Conversely, separate MACs do not fix an IP conflict if both firewalls use the same LAN address. Any other VM NICs that would reach production networks remain disconnected in this mode.
• Emergency means replacement. Once the physical firewall is powered off or completely isolated, the VM assumes the original interface names, the original production IP addresses (from the imported/saved OPNsense configuration), and the original MAC address for each corresponding physical NIC. This makes the replacement look like the old firewall to neighboring devices. It avoids unnecessary changes to DHCP reservations and ISP MAC-based assignments, and minimizes ARP/neighbor-cache disruption. Matching MACs are not intrinsically required by all networks, but cloning them is the most transparent choice for a cold spare.

So we deliberately change the VM's MAC addresses when switching modes. The Proxmox controller applies the MACs specified in the two JSON network profiles; the OPNsense guest does not modify them. Retrieve the production values from ifconfig -a on the physical firewall (look for ether), and generate a unique locally administered MAC for the maintenance client. Never enable the Emergency links while the original firewall is still connected. For a test alongside the physical firewall, use --link-down so all Emergency VM links remain disconnected.

2. Install the guest 90-opnsense-spare hook

Get 90-opnsense-spare from the repository and copy it into the VM, then run:

Code Select Expand

install -m 755 90-opnsense-spare \
  /usr/local/etc/rc.syshook.d/import/90-opnsense-spare
/usr/local/etc/rc.syshook.d/import/90-opnsense-spare init

init saves the current maintenance configuration as the initial maintenance profile. The guest's one import hook then does two things on every boot: read the mode request from the ISO and rename the NICs. It also selects/imports the requested configuration when necessary. No SSH or working network connection is needed for an import, even when all VM NICs have link_down=1.

Note that the initial maintenance configuration may still assign vtnet0. After the first boot with the new hook, that NIC will be called ix1 (or whatever you configured). Use the Proxmox console to correct the maintenance interface assignment/IP/gateway, then verify a second boot. Do not depend on remote access for this first transition.

3. Install the controller on Proxmox

Obtain opnsense-spare and opnsense-spare.json.example from the same repository. On the PVE host:

Code Select Expand

apt install xorriso
install -m 755 opnsense-spare /usr/local/sbin/opnsense-spare
install -m 600 opnsense-spare.json.example /etc/opnsense-spare.json
editor /etc/opnsense-spare.json

The controller changes the VM's Proxmox NIC configuration and attaches a small read-only ISO. That ISO contains the requested mode and interface mapping, and only when explicitly requested a new physical config.xml. The guest performs the actual mode switch during early boot. The ISO has an internal unique request ID, so an ordinary reboot cannot keep reimporting an old config.xml and destroy newer changes.

4. Create and verify the configuration

The JSON describes your VM ID, the network profiles for maintenance and emergency, the mapping of VirtIO names to the original physical names, and the optional SSH/export settings. These are the important parts of the two-NIC example (the actual example file also contains storage, SSH and transfer settings):

Code Select Expand

"interface_names": {
  "vtnet0": "ix1",
  "vtnet1": "igc3"
},
"maintenance_client_nic": "net0",
"networks": {
  "maintenance": {
    "net0": "virtio=02:00:00:00:10:01,bridge=vmbr0,tag=10,firewall=0",
    "net1": "virtio=02:00:00:00:10:02,bridge=vmbr0,link_down=1,firewall=0"
  },
  "emergency": {
    "net0": "virtio=02:00:00:00:20:01,bridge=vmbr0,tag=4094,trunks=1;7;10;52;55;56,firewall=0",
    "net1": "virtio=02:00:00:00:20:02,bridge=vmbr0,tag=41,trunks=40,firewall=0"
  }
}

IMPORTANT: The 02:00:00:... MAC addresses above are illustrative placeholders, not usable defaults. Set networks.emergency.netN to the corresponding physical interface's real MAC and networks.maintenance.net0 to an independent, unique MAC. The maintenance OPNsense configuration also requires an independent LAN IP address; the emergency configuration uses the original firewall's addresses. Replace bridge names, VLAN IDs and paths to match your topology. The script cannot discover them automatically.

Check that the PVE VM already has the expected number of NICs, and that your chosen bridges exist. Then, with the VM stopped, run:

Code Select Expand

opnsense-spare maintenance --dry-run
opnsense-spare maintenance

Start the VM when prompted, fix/check the interface assignment from the PVE console, and verify management connectivity and a repeat boot.

For the first emergency test, keep the real firewall running and disconnect all virtual NIC links:

Code Select Expand

opnsense-spare emergency --link-down \
  --transfer /root/opnsense-transfer/config.xml

Confirm the import and inspect the resulting OPNsense through the PVE console. The first emergency boot needs a config.xml; later you may decline reimporting and reuse the internally saved emergency profile. With --link-down, neither SSH nor any other guest network connectivity is needed.

For a real cutover, shut down the physical firewall and move any necessary cables first. Then use opnsense-spare emergency --link-up, following its prompts. When the physical firewall is repaired, shut down the VM and switch it back to maintenance. An optional SSH export makes the last emergency configuration available on the PVE host for restoring to the physical box.

5. Different network topologies: tag, trunks and physical ports

This is the part you must adapt to your own network. The project is switch-vendor neutral, and the physical OPNsense does not even need to use VLANs.

A PVE VirtIO NIC definition such as:

Code Select Expand

virtio=<MAC>,bridge=vmbr0,tag=4094,trunks=1;10;50

means:

• bridge=vmbr0 is the PVE Linux bridge carrying this VM NIC.
• tag=4094 selects VLAN 4094 as the VM port's untagged/native VLAN (PVID). In this case it is intentionally unused, to avoid putting production untagged traffic onto a live network.
• trunks=1;10;50 permits those VLANs as tagged frames to/from the guest.

My UniFi / VLAN 1 case: My PVE host uses native/untagged VLAN 1 for management on its UniFi uplink, while the physical OPNsense expects VLAN 1 tagged on its LAN trunk. The emergency VM needs to see that VLAN tagged, too. Putting the VM port on a dummy/native VLAN 4094 and allowing VLAN 1 in trunks provides this arrangement on a properly configured VLAN-aware PVE bridge. The physical and PVE uplink port profiles are not necessarily identical. Verify ingress and egress 802.1Q behavior rather than assuming that VLAN 1 is treated the same everywhere.

For the WAN in my particular setup, VLAN 40 carries PPPoE and VLAN 41 transports the ONT's untagged management network across the switch. That explains:

Code Select Expand

virtio=<physical-WAN-MAC>,bridge=vmbr0,tag=41,trunks=40

It is not a universal WAN requirement. If your ISP supplies an untagged DHCP WAN and your PVE host has a spare physical Ethernet adapter, create a separate bridge (say vmbrWAN) connected to that adapter, without assigning a host IP, and use:

Code Select Expand

virtio=<physical-WAN-MAC>,bridge=vmbrWAN

Move the WAN cable to that adapter in an emergency. No WAN transport VLAN or ONT management VLAN is needed.

For a physical firewall with igc0 = untagged LAN, igc1 = tagged LAN VLANs, igc3 = untagged WAN/DHCP, use three VM NICs, three original MACs, and a mapping of vtnet0=igc0, vtnet1=igc1, vtnet2=igc3. The repository includes opnsense-spare-three-nic.json.example. Your PVE bridges can be backed by separate physical NICs or by a suitably configured switch/trunk.

6. Side notes and caveats

SSH agent versus a key file: The initial import does not need SSH. The optional return export is performed later, when maintenance mode is running and reachable over the LAN. You can specify an SSH identity file in the JSON or use "ssh_identity": null and connect to the PVE host with agent forwarding, e.g. ssh -A root@pve-host. The public key must be authorized for the maintenance OPNsense user. Do not disable host-key verification.

UEFI/EFI bootloader: As described above, I recommend OVMF/UEFI for the VM, independently of how the physical firewall boots. I ran into a separate problem after importing a newer physical OPNsense configuration into an older VM installation: a reboot loop during EFI framebuffer/console initialization, before the import hook was even relevant. In that case, updating the VM's EFI loader from its matching OPNsense/FreeBSD installation fixed it. If you encounter this, test serial-console boot and check the ESP and its EFI/BOOT/BOOTX64.EFI and EFI/FreeBSD/loader.efi copies against the installed /boot/loader.efi. Back up the EFI files and identify the correct ESP before changing anything. This is not part of the normal cold-spare procedure and must not be applied blindly.

CPU microcode: If the physical OPNsense uses a CPU microcode plugin, its presence in the configuration is not a problem for the cold spare. The import transfers only the XML, not any microcode binaries. Even if the plugin is installed in the guest, it cannot update the physical CPU through the virtual machine; CPU microcode updates are handled by the Proxmox host. There is therefore no need to remove the plugin from the imported configuration or to treat it as a migration hazard. Installing it separately in the guest serves no useful purpose.

Cold standby, not magic: NIC names and MACs can be reproduced, but physical NIC-specific tunables, plugins, IPS/offloading, interface offload settings and WAN-specific constraints may not transfer cleanly. Test VLANs, DHCP/PPPoE, firewall rules and routing before relying on the VM. Never operate the physical firewall and the emergency VM simultaneously with the same production addresses. Proxmox itself must remain reachable without depending on the failed firewall in an unresolvable way.

Security and disclaimer

I generally recommend not installing untrusted software on security-critical systems such as OPNsense or Proxmox. This project is deliberately straightforward and the source is available for inspection. The guest hook essentially selects/imports a configuration and renames interfaces early during boot. The PVE controller manages VM NIC profiles, packages the mode/configuration files into an ISO, and optionally handles a configuration export. There is no permanently running synchronization agent. The scripts are small enough to audit compared with a full HA stack, and they should not normally need frequent updates.

That said, read the scripts before running them. The ISO and exported config.xml may contain passwords, keys and other firewall secrets: protect them accordingly. A successful unit test is not a substitute for a real disaster-recovery drill.

Everything is provided as-is, without warranties or guarantees of any kind. You are responsible for validating the solution on your own hardware and topology.

This suits my needs - you have to decide if it suits yours.

Your question is waaaay too unspecific. It still goes over port 53 - "it" meaning what, exactly?

What type of DNS service do your clients use? Unbound, DNSmasq, OpenDNS, Bind or something else? How do you know that they do not use something else despite you sending them OpnSense's IP via DHCP, like 8.8.8.8?

Aside from your clients, does your OpnSense itself use upstream or configured servers?

You know, DNS is not as easy as it used to be these days.

Thanks, Cedrik, good point! ECH certainly helps against passive observation of SNI.

However, my concern is active enumeration: An attacker connects to the public proxy using a known SNI name (e.g. www.example.com), then tries different HTTP Host headers (pve.example.com, opnsense.example.com, etc.) over that same TLS connection to discover hidden backends.

As far as I understand, ECH doesn't prevent this, since the attacker is a client, not a passive observer. Or am I missing something?

Placeholder - maybe I will describe a system that can act as a reverse proxy in a DMZ in the future here.

[1. Introduction]

1. Introduction

One of the attractive features of OPNsense is the ability to run additional services directly on the firewall. HAProxy, Caddy and Nginx offer reverse proxy functionality, TLS termination and centralized certificate management through ACME.

Another particularly useful feature is the seamless bridging between IPv4 and IPv6: An IPv6 client can access an IPv4-only backend (and vice versa), without requiring NAT64 or dual-stack support on the backend itself. Running the proxy directly on the firewall makes this especially convenient, since it already has connectivity to both address families and the internal networks.

Squid provides similar convenience for forward proxy functionality. I have been using this approach myself, and it seems only natural to take advantage of these possibilities instead of maintaining additional systems.

However, while reviewing my HAProxy configuration, I realized that there are some serious security implications which had not been entirely obvious to me before. The most important one is a fundamental conflict between running a proxy on the firewall and using that same firewall to enforce network segmentation.

None of this is specific to HAProxy. The architectural problem applies to Caddy, Nginx, Squid and other proxy services as well, although their exposure to attacks obviously depends on how they are deployed.

2. Why Network Segmentation Matters

Consider a typical infrastructure with separate networks or VLANs for management, publicly accessible services (DMZ), application servers, databases, VMs, containers and internal clients. The firewall controls communication between these networks, allowing only explicitly required connections.

The primary purpose is not just to prevent unauthorized access from the Internet. It is to limit the blast radius of a successful compromise by restricting lateral movement between network segments.

In our times of AI-generated attacks, we should assume that sooner or later an Internet-facing application will contain an exploitable vulnerability. If an attacker gains control of a web server, the firewall should prevent that server from becoming a starting point for attacks against the rest of our infrastructure. This is what network segmentation and the principle of least privilege are about.

And this makes sense even in small installations. With Proxmox VE, for example, VMs and LXCs can easily be placed into separate virtual networks, with OPNsense controlling the traffic between them. As long as the segmentation stays within the virtualized environment, not even managed physical switches are required.

3. What Happens When the Proxy Runs on the Firewall?

Consider two architectures:

A: Internet → OPNsense with proxy → Backend services

No question: this is the convenient solution. However, OPNsense permits outgoing connections originating from the firewall itself by default. The automatically generated rule is called "let out anything from firewall host itself". The OPNsense Caddy documentation explicitly confirms that firewall rules allowing the proxy to reach internal services are unnecessary because traffic originating from the firewall is allowed.

Consequently, the firewall has a fundamentally different network position from a server in a DMZ. It generally has routes to all directly connected networks and potentially even others reachable through VPN tunnels. The usual inter-VLAN firewall rules do not automatically restrict connections originating from processes running on the firewall itself.

Squid: No Exploit Required!

With Squid as a forward proxy, the situation can be even worse: No vulnerability is required at all.

In fact, I encountered exactly this problem in my own setup: A client in one VLAN was prohibited by firewall rules from accessing services in another VLAN. However, the same client could reach those services through Squid running on OPNsense!

The reason is simple: Squid establishes the outgoing connections from the firewall itself, effectively bypassing the inter-VLAN restrictions. Unless Squid's own ACLs explicitly prohibit such connections, the network separation is defeated during perfectly normal and intended operation.

This is not a new discovery, either. The problem was already discussed in OPNsense issue #3710 back in 2019.

So with Squid, the security boundary can be bypassed without compromising anything.

Now assume an attacker exploits a remote-code-execution vulnerability in the proxy.

The attacker may be able to initiate connections from the firewall host to systems that would have been inaccessible from a separate DMZ host. This effectively eliminates the security boundary we just created. And the attacker does not necessarily need root privileges for this. An ordinary Unix process can initiate outgoing TCP connections. Running the proxy under an unprivileged account or inside a chroot environment does not, by itself, restrict its network access.

B: Internet → OPNsense → Proxy in DMZ → Backend services

Here the proxy resides in its own network, and OPNsense explicitly permits only the connections it needs to reach its backends. If the proxy is compromised, the attacker can potentially attack those authorized backends, but the firewall continues to block access to everything else.

The security boundary remains intact. That was the whole point of network segmentation in the first place.

Of course, locally generated traffic on OPNsense can also be restricted using pf rules. But the firewall itself needs access for DNS, NTP, updates, certificate management, monitoring, VPNs and numerous other purposes. Building and maintaining suitable restrictions is considerably more complicated than applying a simple policy to a separate DMZ host. Furthermore, destination-based rules do not necessarily distinguish legitimate firewall services from a compromised proxy connecting to the same destination.

So why deliberately create separate security zones only to give a potentially exposed application the network privileges of the device enforcing those zones?

4. Side Aspect: Hiding Administrative Services

There is another, somewhat different issue I encountered while reviewing my HAProxy setup.

Sometimes we want to retain remote access to administrative services without exposing them to routine Internet scans.

A good example is Proxmox VE hosting OPNsense as a VM. If OPNsense fails after an update or configuration mistake, we may need SSH access to the PVE host to repair the firewall or start a fallback VM. Obviously, that access must not depend on the failed OPNsense instance. But we also do not necessarily want to advertise our hypervisor's management interface to the entire Internet.

This requirement is not limited to hypervisors. It applies to various management interfaces and other services that should remain available without being easily discoverable.

There are two interesting approaches I often use:

A: A secret IPv6 address

Assign a randomly generated global IPv6 address to the management interface without publishing it in DNS. Even if an attacker knows the /64 prefix, there are 2^64 (approximately 18.4 quintillion) possible interface identifiers.

Provided the address is genuinely unpredictable, exhaustive port scanning is impractical. This is fundamentally different from IPv4, where scanning an entire subnet is usually trivial. Of course, this assumes that the address is not derived from a MAC address, sequential allocation or another predictable pattern. RFC 7707 discusses ways of discovering IPv6 hosts without brute-forcing the entire address space.

For the PVE example, the host must also have an Internet connection independent of the OPNsense VM.

B: A secret hostname behind a reverse proxy

Alternatively, we could configure an unpublished virtual hostname such as emergency.example.com on our publicly accessible reverse proxy. Only someone knowing that hostname would supposedly reach the corresponding backend. This also handles the problem that arises from the IPv6-only approach: It can be used for both IPv4 and IPv6.

Unfortunately, this approach has a rather unexpected weakness.

5. TLS SNI Is Not the HTTP Hostname

When establishing an HTTPS connection, the client may provide a hostname through TLS Server Name Indication (SNI), which the server uses to select an appropriate certificate. The actual HTTP hostname, however, is transmitted separately, using the Host header in HTTP/1.1 or :authority in HTTP/2.

These names do not necessarily have to match.

Suppose our proxy serves www.example.com publicly, while pve.example.com and opnsense.example.com are unpublished administrative hostnames.

An attacker establishes a TLS connection using the known SNI name www.example.com. Once connected, the attacker can send requests with different HTTP Host headers, such as pve.example.com or opnsense.example.com.

If the proxy routes requests based on those HTTP hostnames without additional restrictions, it will forward them to the respective backends. A single known SNI name can therefore be sufficient to enumerate otherwise hidden virtual hosts. Persistent HTTP connections can carry multiple such requests without requiring a new TLS handshake for each attempt.

And with typical names like pve, proxmox, opnsense, firewall, nas, backup or grafana, the search space is rather small.

5.1. Another Trap: Accidentally Exposing Internal Services

There is another pitfall when using a reverse proxy simply to provide HTTPS and valid certificates for internal services. If these services share a publicly accessible frontend with other applications, they may inadvertently become accessible from the Internet as well.

Neither internal DNS records nor the absence of public DNS entries prevent this. An attacker can connect to the public proxy address and supply the corresponding hostname. This does not even require an SNI/Host mismatch.

Therefore, internal-only services should use a separate frontend bound to internal interfaces or be protected by explicit access restrictions. And remember: ACME DNS-01 validation allows obtaining publicly trusted certificates without exposing the services themselves.

5.2. Forcing Equality of TLS SNI and Hostname?

Why not simply enforce equality between SNI and Host? This is possible, but not always desirable. HTTP/2 explicitly allows connection reuse across different origins under certain conditions. Enforcing strict equality can interfere with legitimate clients. And even if enforced, an attacker can still try matching SNI/Host pairs, one connection at a time.

Of course, a randomly generated hostname with sufficient entropy could be just as difficult to guess as a random IPv6 address. The problem is relying on conventional, predictable but unpublished hostnames as a security measure. Also: Who really uses something like afas4gfhrt76gfg.example.com?

6. Don't Accidentally Reveal Your Secrets

Both approaches rely on information remaining unknown. There are several ways this assumption can fail:

DNS enumeration: Depending on zone configuration, hostnames may be discovered through improperly permitted zone transfers, DNSSEC NSEC enumeration, dictionary attacks or historical DNS records. Certificate Transparency logs may also disclose hostnames for which individual public certificates were issued in the past. Wildcard certificates avoid revealing individual subdomains through their certificates, but not through other mechanisms.

For a "secret" IPv6 address, the obvious precaution is to avoid publishing any corresponding DNS records, including AAAA and revealing reverse DNS (PTR) records.

Outgoing IPv6 connections: There is another easily overlooked problem. A host with multiple global IPv6 addresses may select the supposedly secret management address as the source for outgoing connections. If that happens, the remote server learns the address, which may then appear in logs or other records. Its secrecy is gone, even though it was never published in DNS.

A secret IPv6 management address should therefore not be used as the source for ordinary outgoing Internet connections.

Use a separate address for outgoing traffic and verify the actual source address selection. On Linux, "ip -6 route get" can help identify the selected source for a particular destination. Applications explicitly binding to an address must be considered separately. Temporary IPv6 privacy addresses may help, but should not simply be assumed to solve the problem automatically. Hint: Setting the preferred lifetime of the secret IPv6 address to zero marks it as deprecated, making Linux prefer other suitable source addresses for new outgoing connections (RFC 6724, Rule 3).

Finally, neither a secret IP address nor a secret hostname replaces authentication or access control. Both are additional obstacles against discovery, nothing more.

7. Conclusion

Running HAProxy, Caddy, Nginx or Squid directly on OPNsense is extremely convenient. But convenience comes with architectural consequences.

The problem is not necessarily that the proxy software itself is insecure. The problem is what happens when it becomes compromised.

We deliberately segment our networks to contain such incidents. Placing a potentially exposed proxy directly on the firewall may give an attacker a way around exactly those restrictions.

Similarly, hiding administrative services behind unpublished hostnames may offer considerably less protection against discovery than expected, particularly when a publicly accessible reverse proxy provides a known entry point.

Security is not only about preventing a compromise. It is also about limiting the damage when one occurs. Note that I say "when", not "if" here. And the device enforcing those limits is not necessarily the best place to run the applications that could bypass them.

Thus, all of the available proxies for OPNsense should probably carry a large sign: WARNING: LARK'S VOMIT!!! - but then, our sales would plummet! If you don't know what that means, look it up!

[und]

Gut, ich nehme an, das ist das betroffene Interface-Netz.

Interessant wäre der Vergleich zu den Firewall-Regeln der funktionierenden Interfaces, aber eins sieht man schon jetzt:
Deine "Default allow LAN to any rule" erlaubt bei Dir nur spezifischen TCP-Traffic. Du hast offenbar die Default-Regel für das erste LAN angepasst, ohne die Bezeichnung zu ändern.

Am Rande bemerkt: Je nachdem, zu welchem Zweck man VLANs anlegt, schottet man diese normalerweise gegeneinander ab. Falls die Segmentierung ausschließlich der Trennung von Broadcast Domains zur Vermeidung von zu viel Broadcast Traffic dient, gilt das natürlich nicht.
Will sagen: Die normale "Default allow LAN to any rule" ist sinngemäß sinnvoll für kleine Heimnetze ohne VLANs, mit VLANs würde man zuvorderst mal die anderen (V)LANs schützen und trotzdem (bestimmten) Internet-Verkehr zulassen - es gibt einige Threads dazu, u.a. wie man limitierte Netze zu Gruppen zusammenfasst, um das zentral zu regeln.

Bei Deiner Konfiguration geht mehr (nämlich Zugriff auf andere VLANs)  und weniger (z.B. kein Ping) als das, also würde ich annehmen, dass sie noch weitere Fragen aufwirft. Beispielsweise ist nicht nur Ping betroffen, wenn Du explizit nur bestimmte TCP-Ports erlaubst: Wie soll dann z.B. DNS funktionieren und andere Firewall-Services?

Ich will nichts unterstellen, aber vielleicht liest Du zuerst mal dies - oder sogar dies, je nachdem, was Du vorhast.

If only Mikrotik configuration was not such a hassle... I like Unifi for everything switching and APs.

Having done 802.1x for ~200.000 clients in an enterprise environment, I think the real benefits are unachievable in a private environment:

You need RADIUS-capable switching equipment for that (implemented correctly, unlike with some Unifi switches in the recent past) and also, MAC-based 802.1x can certainly be circumvented. Some client devices do not even work with that.

On the other hand, "secure" certificate-based 802.1x is hard to handle and you need client OS support, which many IoT devices essentially lack.

The situation is a little better for WiFi, since it has inherent access limiting capabilities.

But @Patrick: device isolation is useless in many contexts, IMHO: think of media sharing and broadcast-based traffic. I just had a Miele hob not contacting its extractor hood because of client isolation. Also, it does not even work over different APs, only for devices connected to the same AP.

How about the official documentation for a start?

[multiple gateways and policy-based routing (PBR)]

Your latest tcpdump provides an important clue: Even in the broken state, DNS queries from 10.127.127.2 reach 10.127.127.1, and valid DNS responses are sent back through wg4.

This suggests that Unbound is working. The problem is more likely related to packet delivery, firewall rules, or routing/gateway selection during startup.

Since your setup involves multiple gateways and policy-based routing (PBR), I particularly suspect a routing or firewall state issue during boot, possibly involving incorrect gateway selection or asymmetric routing.

Could you perform the following tests before restarting WireGuard, while simultaneously running `tcpdump -ni wg4 'host 10.127.127.2'` on OPNsense?

1. Explicit DNS lookup: Query `10.127.127.1` directly from the affected client (e.g. `nslookup example.com 10.127.127.1`). If this fails despite DNS responses appearing in the capture, the problem is likely on the return path to the client or in the client's DNS stack. If it succeeds, DNS transport itself is working.

2. Direct IP connection: Try accessing a known reachable Internet server by IP address, bypassing DNS entirely. If this fails while DNS works, suspect forwarding, NAT, firewall rules, or gateway selection. The capture will show whether the connection attempt reaches wg4.

3. Connection by hostname: Access the same server by hostname. If direct IP access works but hostname access fails, investigate the client's DNS configuration or resolver behavior rather than general IP routing.

Given your multiple gateways and PBR rules, I would also compare the firewall states and gateway selection for the affected client traffic before and after restarting WireGuard.

These tests should distinguish DNS transport issues from forwarding/routing problems and incorrect client-side name resolution.

I would avoid introducing startup delays until we have identified the failing component.

Generally, there are usually one of these problems (not all with all DHCP servers, but hey):

1. With some servers (at least ISC), the creation of a static reservation does not get active unless the service is restarted.
2. Some servers (also ISC), you should not have static reservations within dynamic ranges.
3. When you let a client get a dynamic reservation and convert it to a static one, the dynamic reservation is not deleted, such that a reboot still gives the old IP. You have to manually clean the dynamic reservation from the lease file (and restart the service to recognize that fact).
4. With newer services, a DUID ist being used instead of a MAC (unless you change that setting) - which may give the impression that static reservations (for the MAC) do not work.

That being said, I know that is not an exact answer, just because I happen to use Kea and not DNSmasq.