Messages

1

24.7 Production Series / Re: DNS Over TLS Broken

« on: November 22, 2024, 07:07:57 pm »

DoT works for me - also with Quad9.

Your dump of the local connection looks fine - exactly like mine. So, if your unbound cannot handle the SSL connection with a "error: ssl handshake cert error: unable to get local issuer certificate" message, it seems that its certificate chain is off.

I would think that something in your trust settings must be off, although I do not see why the console would work and unbound does not.

I would check system health if there are altered files or a defective file system.

2

German - Deutsch / Re: Glasfaser Einrichtung / OpnSense / Modem notwendig?

« on: November 22, 2024, 01:26:39 pm »

Das geht theoretisch - da muss normalerweise aber der ISP mitspielen, dass er die ONT-ID bei sich einträgt. Normal ist nur die ID des mitgelieferten ONT hinterlegt. Wegen Endgerätefreiheit müssen die ISPs das zwar zulassen, tun es aber sehr ungern.

Das ist auch insofern problematisch, weil Du so kein Backup hast (Highlander-Regel: es kann nur einen geben). Besser klappt es, wenn man die ID des originalen ONTs auslesen kann - das geht bei DG aber, je nach Ausbaugebiet und verbauter Technik, eventuell nicht.

Noch ein Nachteil dabei ist: Support macht DG nur für die eigene Technik.

Das ist m.E. den Stress nicht wert.

3

German - Deutsch / Re: Glasfaser Einrichtung / OpnSense / Modem notwendig?

« on: November 22, 2024, 12:11:21 pm »

Prinzipiell ja, allerdings weiß ich nicht, wie genau die Business-Anschlüsse von DG sich unterscheiden.

Was mich ein bisschen wundert: Es gibt bei DG zwei Sorten Business-Anschlüsse - DG professional und DG Business. Letzterer ist der "große" Anschluss, bei dem es auch routebare IPv4 gibt, was bei allen anderen Tarifen nicht der Fall ist.

Zu dem großen Tarif sind kaum Unterlagen zu finden - weder Preise noch Anschlussinfos. Bei den Privatanschlüssen gibt es Varianten "mit eigenem Router", die man wählen sollte, wenn man die Fritzbox der DG nicht nutzen will. Das ist m.W. Voraussetzung dafür, dass man überhaupt einen eigenen Router einsetzen kann.

Die Telefonie bei den Privatanschlüssen ist einfach SIP, also direkt in der nachgelagerten Fritzbox konfigurierbar.

Bei DG professional scheint es so zu sein wie beim Privatkundenanschluss, bis auf den Preis.


Ich weiß für DG Business überhaupt nicht, wie sie da verfahren. Das geht alles nur mit direktem Kontakt, keine Downloads zu Specs, keine Preise, kein Nichts. Wirst Dich überraschen lassen müssen. Darf man fragen, was der Anschluss kostet (bei welcher Geschwindigkeit)?

4

German - Deutsch / Re: Glasfaser Einrichtung / OpnSense / Modem notwendig?

« on: November 22, 2024, 11:21:49 am »

Lies mal: https://forum.opnsense.org/index.php?topic=39556, das beantwortet die Frage nach der Fritzbox "vor" der OpnSense. Kurz gesagt: besser nicht.

Du hast nicht geschrieben, welcher ISP das ist. Je nach ISP machen die mal mehr, mal weniger Stress mit dem Einsatz eigener Router. Normalerweise sollte es nur eine Ethernet-Verbindung vom OpnSense WAN zum vom ISP gelieferten ONT sein, meist per DHCP konfiguriert. Hängt aber vom Provider und der Anschlussart ab.

5

General Discussion / Re: ESTABLISHED,RELATED - How to block NEW ?

« on: November 21, 2024, 05:00:12 pm »

There is a sequence number, correct, but since you can "smoke ping" and there is no foreseeable sequence in which packets get transmitted, you cannot decide based on the sequence number (or, in other words: even if you used something akin to a "catch range", it would be just as good as an abitrary time range).

6

General Discussion / Re: ESTABLISHED,RELATED - How to block NEW ?

« on: November 21, 2024, 03:57:44 pm »

However, my issue is the following when one does step by step :
1. first, A pings B : there is no answer - correct
2. second, B pings A : it works - correct
3. but now, if A pings B, A gets replies - NOT CORRECT

You seem not to understand that - unlike with real stateful traffic like TCP - with ICMP traffic, there is no "state" other than a time period where traffic in the reverse direction was seen.

A established TCP connection is characterized by a quadruple of (src_ip, src_port, dst_ip, dst_port). Thus, you can actually determine which response packets are to be allowed. If a packet was observed with a different dst_port as sender, it would not be allowed, since it cannot be attributed to the existing connection.

With ICMP, this is different. There is no port, only an ICMP subtype. Other than that, there are only (src_ip, dst_ip). Thus, you can only decide based on "soft" factors ("related"), like if within the last few seconds, you saw ICMP traffic between both parties that might explain why another ICMP packet is seen (and being passed).

Try the same test on an iptables-based firewall and you will probably see the same result.
Or, retry step 3 of your test after some time has passed.

7

German - Deutsch / Re: OPNsense verbindungs Issues

« on: November 21, 2024, 01:35:01 pm »

Die Idee, die Netzwerkkarte durchzureichen, kann - je nach Anwendungsfall - nicht so gut sein, speziell, wenn der FreeBSD-Treiber problematisch ist.

8

24.7 Production Series / Re: 24.7.7 - Not responding to IPv6 Neighbor Advertisments

« on: November 21, 2024, 01:31:02 pm »

You can edit the thread title. BTW: Did you see this?

9

24.7 Production Series / Re: 24.7.7 - Not responding to IPv6 Neighbor Advertisments

« on: November 21, 2024, 01:18:47 pm »

It's still going strong with the Linux bridge config! I think I'll just leave it like this now, because I don't seem to be having any problems. I wonder if maybe the Realtek NICs were a problem before?

FreeBSD Realtek drivers have been known to cause problems in the past. By using them bridged, you do not have to rely on FreeBSD drivers. Matter-of-fact, the vtnet drivers are often more performant than FreeBSD native drivers.

10

Tutorials and FAQs / Re: [HOWTO] OpnSense under virtualisation (Proxmox et.al.)

« on: November 21, 2024, 10:44:18 am »

Caveat, emptor: This is unfinished!



Setup for OpnSense and Proxmox for a datacenter

A frequently used variant is to work with two bridges on Proxmox:

• vmrb0 as a bridge to which Proxmox itself, OpnSense WAN interface and VMs with a separate IP can connect (even if you don't use it)
  
• vmbr1 as a LAN or separated VLANs from which all VMs, OpnSense and Proxmox can be managed via VPN
  

That means you probably need two IPv4s for this setup. You should also get at least a /56 IPv6 prefix, which you need for SLAAC on up to 256 different subnets.

While it is possible to have just one IPv4 for both OpnSense and Proxmox, I would advise against it. You would have to use a port-forward on Proxmox, which results in an RFC1918 WAN IPv4 for OpnSense, which in turn has implications on NAT reflection that you would not want to deal with.


The configuration then looks something like this:

Code: [Select]

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto eth0
iface eth0 inet manual
iface eth0 inet6 manual

auto vmbr0
iface vmbr0 inet static
        address x.y.z.86/32
        gateway x.y.z.65
        bridge-ports eth0
        bridge-stp off
        bridge-fd 0
        bridge-mcsnoop 0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
        post-up echo 1 > /proc/sys/net/ipv6/conf/eth0/forwarding
        #up ip route add x.y.z.76/32 dev vmbr0
        #up ip route add x.y.z.77/32 dev vmbr0
#Proxmox WAN Bridge

iface vmbr0 inet6 static
        address 2a01:x:y:z:5423::15/80
        address 2a01:x:y:z:87::2/80
        address 2a01:x:y:z:88::2/80
        address 2a01:x:y:z:89::2/80
        address 2a01:x:y:z:172::2/80
        gateway fe80::1
        post-up ip -6 route add 2a01:x:y:f600::/64 via 2a01:x:y:z:172::1

auto vmbr1
iface vmbr1 inet static
        address 192.168.123.2/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        bridge-mcsnoop 0
        post-up ip route add 192.168.0.0/16 via 192.168.123.1 dev vmbr1
#LAN bridge

iface vmbr1 inet6 static

source /etc/network/interfaces.d/*



This includes:

x.y.z.86 main Proxmox IP with x.y.z.65 as gateway,

x.y.z.87 WAN IPv4 of OpnSense,

x.y.z.88 and x.y.z.z.89 additional IPs on vmbr0. These use x.y.z.86 as gateway so that your MAC is not visible to the ISP. Hetzner, for example, would need virtual MACs for this.

192.168.123.2 is the LAN IP for Proxmox so that it can be reached via VPN. The route is set so that the VPN responses are also routed via OpnSense and not to the default gateway.

IPv6 is a little more complex:

2a01:x:y:z:: is a /64 prefix that you can get from your ISP, for example. It is further subdivided with /80 to:

2a01:x:y:z:1234::/80 for vmbr0 with 2a01:x:y:z:1234::15/128 as external IPv6 for Proxmox.
2a01:x:y:z:172::15/128 as point-to-point IPv6 in vmbr1 for the OpnSense WAN with 2a01:x:y:z:172::1/128.

2a01:x:y:z:124::/80 as a subnet for vmbr1, namely as an IPv6 LAN for the OpnSense.

The OpnSense thus manages your LAN with 192.168.123.1/24 and can do DHCPv4 there. It is the gateway and DNS server and does NAT to the Internet via its WAN address x.y.z.87. It can also serve as a gateway for IPv4 with the IPv6 2a01:x:y:z:123::1/64.

VMs would have to get a static IPv6 or be served via SLAAC. That only works with a whole /64 subnet. The prefix, 2a01:x:y:rr00::/56, is used for this, which can then be split into individual /64 prefixes on the OpnSense and distributed to the LAN(s) via SLAAC (e.g. Hetzner offers something like this for a one-off fee of €15).

You can use the additional IPs, but you don't have to. These "directly connected" VMs could, for example, also use IPv6 in 2a01:x:y:rr00::/64.

Some more points

1. You can/should close the Proxmox ports, at least for IPv4, of course, but you can still keep them accessible via IPv6. This means you can access the Proxmox even without OpnSense running. There is hardly any risk if nobody knows the external IPv6, as port scans in IPv6 hardly seem to make sense. But be careful: entries in the DNS could be visible and every ACME certificate is exposed, so if you do, only use wildcards!

2. I would also set up a client VM that is located exclusively in the LAN and has a graphical interface and a browser and that is always running. As long as the Proxmox works and its GUI is accessible via port 8006, you have a VM with LAN access and a browser. This also applies if the OpnSense is messed up and no VPN is currently working. The call chain is then: Browser -> https://[2a01:x:y:z:1234::15]:8006, there is a console to the client VM, there access https://192.168.123.1/ (OpnSense LAN IP) with the browser.

3. Be careful with (asymmetric) routes! Proxmox, for example, has several interfaces, so it is important to set the routes correctly if necessary. Note that I have not set an address for IPv6 on vmbr1 because it is actually only intended to be used for access via VPN over the LAN. However, if the OpnSense makes router advertisements on the LAN interface, you quickly have an alternative route for Proxmox...

4. You can use fe80::1/64 as virtual IPs for any (V)LAN interface on OpnSense. That way, you can set fe80::1 as IPv6 gateway for the VMs.


VPN

It is up to your preference on which VPN you should use to access the LAN or VLANs behind your OpnSense. I use Wireguard site to site.
There are tutorials on how to do this, but as an outline:

• Choose a port to make the connection and open it.
  
• Set up the Wireguard instance to listen on that port.
  
• Connect  a peer by settings the secrets.
  
• Allow the VPN traffic (but wisely!)
  
• Check the routes if you cannot reach the other side.
  

VLAN setup

In order to isolate traffic between the VMs, you can also choose to have vmbr1 to be VLAN-aware. In that case, you will have to assign each VM a separate VLAN, define VLAN interfaces on OpnSense and break up small portions of the RFC1918 LAN network to use at leat 2 IPv4s for OpnSense and the specific VM.
You can do the same with your IPv6 range, because you have 256 IPv6 prefixes - so each VM can have its own /64 range and could even use IPv6 privacy extensions.
Since OpnSense is the main router for anything, you will still be able to access each VM via the VPN by using rules for the surrounding RFC1918 network.


Reverse proxies

If you want to make use of your OpnSense's capabilities, you will have to place your VMs behind it, anyway. If you are like me, and want to save on cost for additiional IPv4s, you can make use of a reverse proxy.

On HAProxy vs. Caddy (there is a discussion about this starting here):

Today I took the opportunity to try out Caddy reverse proxy instead of HAproxy, mostly because of a very specific problem with HAproxy...

I must say I reverted after trying it thoroughly. My 2cents on this are as follows:

- Caddy is suited to home setups and inexperienced users. HAproxy is much more complex.
- For example, the certificate setup is much easier, because you just have to specify the domain and it just works (tm).
- However, if you have more than just one domain, Caddy setup gets a little tedious:
* you have to create one domain/certificate plus a http backend for any domain, which includes creating different ones for www.domain.de and domain.de. You cannot combine certificates for multiple domains unless they are subdomains.
* You do not have much control over what type of certificate(s) are created - you cannot specifiy strength or ECC vs. RSA (much less both) and I have not found a means to control if ZeroSSL vs. LetsEncrypt is used.
* The ciphers being employed cannot be controlled easily - or, for TLS 1.3, at all. That results in an ssllabs.com score which is suboptimal, becaus 128bit ciphers are allowed. This cannot be changed because of Go limitations.
* You cannot use more than one type of DNS-01 verification if you use wildcard domains.
* The Auto HTTPS feature looks nice first, but indeed it uses a 308 instead of a 301 code, which breaks some monitoring and can only be modified via custom include files.

So, if you just want to reverse-proxy some services in your home network, go with Caddy. For an OpnSense guarding your internet site with several services/domains, stay with HAproxy.

There are nice tutorials for both HAproxy and Caddy, so use them for reference.

11

Tutorials and FAQs / [HOWTO] OpnSense under virtualisation (Proxmox et.al.)

« on: November 21, 2024, 10:43:58 am »

These days, there are many folks who use OpnSense under a virtualisation host, like Proxmox, for example.

This configuration has its own pitfalls, therefore I wanted to have this guide. The first part starts with common settings needed, the second part will deal with a setup where the virtualisation host is to be deployed remotely (e.g. in a datacenter) and holds other VMs besides OpnSense.

RAM, CPU and system

Use at least 8 GByte, better 16 GBytes of RAM and do not enable balloning. Although OpnSense does not need that much RAM, it can be beneficial in case you put /var/log in RAM (see below).

Obviously, you should use "host" CPU type in order not to sacrifice performance by emulation. However, you should not install the microcode update packages in OpnSense - they would be useless anyway. Instead, install the appropriate microcode packages on the virtualisation host.

The system architecture is arbitrary, as OpnSense can boot both in legacy (BIOS) or UEFI mode.

Filesystem peculiarities

First off, when you create an OpnSense VM, what should you choose as file system? If you have Proxmox, it will likely use ZFS, so you need to choose between UFS and ZFS for OpnSense itself. Although it is often said that ZFS underneath ZFS is more overhead, I would use it regardless, just because UFS fails more often.

32 GBytes is a minimum I would recommend for disk size. It may be difficult to increase the size later on.

After a while, you will notice, that the space you have allocated for the OpnSense disk will grow to use 100%, despite that within OpnSense, the disk may be mostly unused. That is a side-effect of the copy-on-write feature of ZFS: writing logs and RRD data and other statistics always writes new data and the old data does not get dismissed against the underlying (virtual) block device.

That is, if the ZFS "autotrim" feature is not set manually. You can either set this via the OpnSense CLI with "zpool set autotrim=on zroot" or, better, add a daily cron job to to this (System: Settings: Cron) with "zroot" as parameter.
You can trim your zpool once via CLI with "zpool trim zroot".

That being said, you should always avoid to fill up the space for the disk by having verbose logging. If you do not need to keep your logs, you can also put them on a RAM disk (System: Settings: Miscellaneous).

Network "hardware"

With modern FreeBSD, there should not be any more discussion about pass-through vs. emulated VTNET adapters: the latter are often faster. This is because Linux drivers are often more optimized than the FreeBSD ones. There are exceptions to the rule, but not many.

In some situations, you basically have no choice than to use vtnet anyway, e.g.:

• If FreeBSD has no driver for your NIC hardware
• If the adapter must be bridged, e.g. in a datacenter with a single NIC machine

Also, some FreeBSD drivers are known to have caused problems in the past, e.g. for RealTek NICs. By using vtnet, you rely on the often better Linux drivers for such chips.

With vtnet, you should make sure that hardware checksumming is off ("hw.vtnet.csum_disable=1", which is the default on new OpnSense installations anyway).

When you use bridging with vtnet, there is a known Linux bug with IPv6 multicasting, that breaks IPv6 after a few minutes. It can be avoided by disabling multicast snooping in /etc/network/interfaces of the Proxmox host like:


auto vmbr0
iface vmbr0 inet manual
    bridge-ports eth0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
    bridge-mcsnoop 0


TL;DR

• Have at least 8 GByte of RAM, non-balooning
• Use "host" type CPU
• Use ZFS, dummy
• Keep 20% free space
• Add a trim job to your zpool
• Use vtnet
• Check if hardware checksumming is off on OpnSense
• Disable multicast snooping

That is all for now, recommendations welcome!

12

24.7 Production Series / Re: 24.7.7 - Not responding to IPv6 Neighbor Advertisments

« on: November 21, 2024, 10:03:20 am »

I would use vtnet, because it is faster than the emulation these days and also would disable multicast snooping for good measure. In my experiments, it always worked for a few minutes and then failed.

13

24.7 Production Series / Re: 24.7.7 - Not responding to IPv6 Neighbor Advertisments

« on: November 21, 2024, 12:27:29 am »

Is this on a VM by any chance? There is a known IPv6 multicast problem in Linux networking (see here).

14

24.7 Production Series / Re: 24.7.9 - Can't login WebGUI with 2FA

« on: November 21, 2024, 12:16:14 am »

No, I did not set the "always reboot". I only occured to me because with all of the other boxes, where 2FA was disabled, I could upgrade via web UI and was not told that an upgrade were imminent, whereas here, it was different.

I could not do a reboot at that time, so I delayed the update until later, only to find that there was no reboot at all.

So I assume that there is a difference in how the web UI detects if a reboot is neccessary vs. the CLI.

15

24.7 Production Series / Re: 24.7.9 - Can't login WebGUI with 2FA

« on: November 20, 2024, 05:48:00 pm »

I stumbled over that one, too. However, there is another problem:

Once you try to upgrade via ssh, you will be told:

  0) Logout                              7) Ping host
  1) Assign interfaces                   8 ) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Update from console
  6) Reboot system                      13) Restore a backup

Enter an option: 12

Fetching change log information, please wait... done

This will automatically fetch all available updates and apply them.

This update requires a reboot.

Proceed with this action? [y/N]: y


Neither the original upgrade from 24.7.8 to 24.7.9 nor the second upgrade to 24.7.9_1 needed a reboot. I think that the notification on the CLI upgrade is wrong about needing that.