Messages

Hi there, is anyone here running OPNsense on a Hetzner CX... VM ?
For a couple of weeks I am dealing with a highly unstable setup which will fail within hours or sometimes days.
What I need is OPNsense+ReverseProxy with CrowdSec and Wireguard to support the service. I meanwhile got rid of Crowdsec and Wireguard to limit the moving parts but still, nginx or sometimes the OPNsense UI will freeze after some time.

When things go bad I can still get to the console and here's what I noticed:
Sometimes there's the following error: "Listen queue overflow: 193 already in queue awaiting acceptance"
I have found several entries in this forum, suggesting to turn off IPV6. But unless I forgot a v6 setting somewhere, it didn't help.

I tried to restart nginx via service nginx stop/restart but usually this will just show the process id and get stuck.

The only way to get the service back is to reboot. But reboots often get stuck as well because not all services will stop, might be the same nginx service. The console will then show this error: "some processes would not die, ps axl advised" and stay there until I do a power cycle.

With a physical machine I would start to consider hardware or compatibility issues, but a VM from a large hoster?

Edit:
After having been close to abandon this project I decided to start all over with a new VM. Well, what can I say, new VM, restored backup from faulty VM and got a stable running system with nginx+wireguard+CrowdSec active.
I don't know what issue the first VM had but whatever it was is gone now...

Hallo zusammen, ich bin auch gerade dabei mein Homelab zu Hetzner umzuziehen, und wie zu erwarten wenn ich hier schreibe, gibts ein Problem.
Und zwar kann mein LAN Client weder die OPNsense noch irgend eine externe Adresse erreichen.
Zuerst vorneweg, ich bin kein Netzwerkprofi, aber das ist auch nicht die erste OPNsense die ich installiere.

Derzeit habe ich 4 CPX11/21 am laufen, eine davon ist die OPNsense, und sobald alles funktioniert sollen die anderen die momentan im offenen Internet hängen hinter die Firewall.
Nachdem ich den Thread hier und die Hetzneranleitung zu pfsense, ein paar mal durchgelesen habe ist die Situation wie folgt:

Ich habe in der Hetzner Cloud Console ein Netzwerk mit den folgenden Werten (Standardeinstellungen) eingerichtet:
- Netz: 10.0.0.0/16
- Subnetz: 10.0.0.0/24
- Eine Route mit Ziel 0.0.0.0/0 und Gateway 10.0.0.2 (die LAN Adresse der OPNsense)

Weiterhin habe ich eine Debian VM eingerichtet die nur im LAN hängt, IP 10.0.0.4/32 und via "ip route add default via 10.0.0.1" eine Default Route eingerichtet.
Laut Hetzner Dokument brauche ich auf dem Client eine Route zum Hetzner Gateway, und auf dem Hetzner Gateway eine Route zur OPNsense. Ich denke das habe ich mit den beiden obigen Einstellungen erledigt?

Der Debian Client kann via ping alle IPs bis auf die OPNsense im 10.0.0.0 Netz erreichen. Da scheint also mit dem LAN Gateway noch was zu klemmen?
Traceroute zu einer beliebigen Adresse außerhalb geht genau bis zum 1. Hop, dem Hetzner Gateway auf 10.0.0.1, und dann ist Schluß. Das heißt dann wohl die Default Route wird korrekt genutzt, aber das Hetzner Gateway kann nicht mit der OPNsense kommunizieren.

Wenn ich auf der Console der OPNsense bin kann ich mit einem Ping sowohl im LAN als auch im WAN alle Adressen erreichen. DNS funktioniert auch. SSH über das WAN funktioniert auch. Interessanterweise komme ich mit einem ping auch an die 10.0.0.4, den LAN Client, der seinerseits die OPNsense nicht erreicht?

Das LAN Routing auf der OPNsense scheinen mir auch zu passen?
root@OPNsense:~ # netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.31.1.1         UGS      vtnet0
10.0.0.1           link#2             UHS      vtnet1
10.0.0.2           link#2             UH          lo0
...

Nachdem ich hier feststecke habe ich versucht sämtliche Vorschläge hier im Thread nachzuvollziehen.

1. Nach der Basisinstallation habe ich mir, wie in Antwort 2 vorgeschlagen, gleich eine Regel hinzugefügt um (ausschließlich) meinem Rechner WAN Zugriff auf das Webinterface zu geben:
Firewall/Rules/WAN:
Protocol    Source      Port    Destination    Port    Gateway    Schedule       Description    
IPv4*       Meine IP  *    This Firewall    *        *            *               Access for just my IP to the WebUI
Funktioniert!

2. Weiterhin wird hier vorgeschlagen einen Alias anzulegen. Habe ich gemacht:
Firewall/Aliases:
Enabled Name                   Type            Description   Content           Loaded#
x           HetznerNetAlias     Network(s)                10.0.0.0/24      1   
Wobei mir nicht klar ist wo ich den Alias dann benutzen muss?

3. Danach habe ich mich an den Vorschlag von JeGr in #13 gehalten und auf der OPNsense das LAN Interface via DHCP anstatt statisch eingerichtet, sowie eine statische Route zu 10.0.0.0/24 eingerichtet.
System/Routes/Configuration/
Disabled  Network        Gateway                 
         10.0.0.0/24     LAN_DHCP - 10.0.0.1       

Was mir nicht klar ist ist die Aussage von JeGr hier:
<Man muss lediglich aufpassen, dass man auf dem LAN dann nicht versehentlich "LAN network" verwendet oder verwenden will, da das dann /32 wäre und nichts bringt. Man muss also mit dem Alias 10.0.0.0/24 arbeiten das man sich manuell anlegt>>
Wo muß ich mit dem Alias arbeiten?

4. Dann habe ich noch wie in #15 vorgestellt das LAN Standardgateway deaktivert
System/Gateways/Single das LAN_DHCP Gateway deaktiviert, die beiden WAN_DHCP und WAN_DHCP6 waren schon auf active geschaltet, und das WAN_DHCP als Upstream eingestellt.
Name                   Interface       Protocol     Priority                    Gateway            Status   Description
WAN_DHCP (active)    WAN        IPv4        254 (upstream)    172.31.1.1    Online   Interface WAN_DHCP Gateway    
WAN_DHCP6 (active)    WAN        IPv6        254 (upstream)              Online   Interface WAN_DHCP6 Gateway
LAN_DHCP            LAN        IPv4        254            10.0.0.1    Pending  Interface LAN_DHCP Gateway <- Ausgegraut/Deaktiviert

Komischerweise ist LAN_DHCP zwar ausgegraut, aber der Status bleibt selbst nach einem Reboot auf Pending

5. Dann gab's in #18 noch den Vorschlag in Firewall/NAT/Outbound auf Hybrid zu stellen, sowie die Regel unten anzulegen
Interface    Source            Source Port    Destination    Destination Port    NAT Address        NAT Port    Static Port
WAN        10.0.0.0/16     *                *                *                    Interface address    *            NO

Sollte hier als Source eventuell der Alias stehen? Und warum /16 und nicht /24? ich habs ausprobiert, hat nichts genutzt.

Und jetzt bin ich mit meiner Weisheit am Ende, eventuell klemmts ja gar nicht an der OPNsense sondern dass das Hetzner Gateway nicht funktioniert, aber da gibt es eigentlich außer den 3 Punkten ganz am Anfang nichts einzustellen.
Fällt eventuell jemandem hierzu was ein?

Vielen Dank schon mal...

From reading various threads here I understand that os-wireguard_go is an "old" incarnation of the os-wireguard module. The later one is active on my server, but for some reason System/Firmware/Plugins show os-wireguard_go as well but flagged as missing. The only option I got in the UI is to install the mising os-wireguard_go plugin which, as expected, removes os-wireguard. When I then install os-wireguard again, os-wireguard-go shows as missing again?

Given that I can only have one of the two, is there a way to clean up the list pf plugins and to remove the missing wireguard-go plugin? Maybe cli level?

Thanks

I had a somewhat different issue. After upgrading to 23.1.6 all machines but one were working just fine.

The machine which ran into problems was my docker host (Proxmox container) which all of a sudden wasn't able to communicate with it's default gateway = OPNsense. A ping to this particular IP would time out, pinging all other IPs on the same network worked fine. Initially this issue looked completely unrelated to the upgrade, because all other machines on the network, including the Proxmox container host, were able to communicate just fine.

After quite a few hours of unsuccessful troubleshooting I finally ran out of ideas and considered, what I thought is a long shot, restoring OPNsense to the previous version. Restore complete and voila, the container was able to communicate to the external world again!?!?

For now I'll stay on 23.1.5...need to find some time to look into this in some more detail

Great! Thanks a lot.
I ended up applying a backup in the meantime but this might come in handy in the future.

Is there a cli way of changing which cert is used for the admin UI?
I seem to have locked myself out by accidentally selecting the wrong cert. Browser runs into timeout.
System" - "Settings" - "Administration". The Listbox under "SSL certificate"

I went through all the updates from v22 to 23.1 last night and at first glance things looked fine. However this morning we noticed that the haproxy service isn't running.
Unfortunately the log isn't at all telling. All it says is the below:
   2023-02-22T10:06:00   Error   configd.py   [eacd69f9-4ff4-433b-90ed-bd00cb64b4a7] returned exit status 1

The ID seems to be random as it changes with every start attempt.
After searching for similar cases here, I also tried to manually start the service from the cli, unfortunately I am not any wiser:

root@opn2:~ # service haproxy start
Starting haproxy.
/usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy

Is there any way to get more information on why it won't start?
I think I noticed there were some Python3 (?) warnings when the update ran, but the messages went by too quickly to read them. Are the upgrade logs still available somewhere by any chance?

Any help is highly appreciated
Thanks

Just to close this off, I think I got to the bottom of my issue. Indeed not related to OpnSense but rather the cert settings of a server behind NAT.
Once I knew where to look I found this command which allowed me to identify the server which responded with the wrong certificate:

openssl s_client -starttls smtp -showcerts -connect out.mydomain.com:25 -servername out.mydomain.com

Thanks for helping me along!

Thanks again Vilhonator, I am not sure if my issue is specific to email, I am setting up an email relay which technically is a simple network connection protected by TLS. I don't see how any of the higher level email checks would be relevant here. But nevertheless, yes DMARC, SPF and DKIM are all set for this domain.
I actually use both of these FQDNs for relaying email from/to Office365, and that works. Maybe because Microsoft doesn't check the cert.

I attached a screenshot which shows the Gmail dialogs side by side.
Both mail.mydomain.com and out.mydomain.com point to the same server, the only difference (I am aware of) is different IP addresses/interfaces where OpnSense picks up the traffic.
And my question comes down to if the cert gets picked up for mydomain.com what could prevent OpnSense from providing the same cert to out.mydomain.com

You raised a valid point that out.mydomain.com may not be listed in the cert, but given that the cert contains *.mydomain.com, all subdomains should be served, and it does seem to work for mail.mydomain.com.

Let me rephrase the questions slightly: is adding a certificate to the OpnSense trust store sufficient for it to be used for all FQDNs listed in the certificate, or do I need to specifically assign the certificate to a WAN interface/NAT rule/ anything? If the answer is that this certificate will be used for all services, I may be hitting the wrong bush. E.g. OpnSense provides the correct cert but Gmail for some reason is not happy with it which would mean I can tick off OpnSense and need to continue troubleshooting at that end.

Thanks for joining in Vilhonator, that's a good idea, it didn't occur to me to check if the individual FQDN are listed in the cert.
But unfortunately they are not, not for the working mail.mydomain.com nor for the failing out.mydomain.com.

Here's what the respective section looks like when I select info for the certificate in OpnSense:

X509v3 Extended Key Usage:
               TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
X509v3 Subject Alternative Name:
                DNS:*.mydomain.com, DNS:mydomain.com

I guess that means this certificate should get handed out for all connection requests, no matter the subdomain?
The certificate is issued by certum.pl, not Letsencrypt.

One other idea I had was that maybe the cert is handed out by OpnSense at all but by one of the services NAT redirects to, in my case that's a Proxmox Mail Gateway. But this server hasn't got the certificate loaded so I wouldn't know how it gets injected.

<<All I had to do to get certificates was to add domain key provided by my mail server provider to DNS records.>>
That's for the dkim keys I think? My issue is related to establishing a TLS connection between Gmail and my server,

Hi there, I have inherited a 2 server OpnSense cluster setup (21.7.6-amd64) which is humming nicely for more than a year, but now that I need to connect a new service I ran into an issue I struggle to figure out myself.

My setup exposes two IPs per server (one physical/ one virtual) which are tied to their associated FQDN = mail.mydomain.com and out.mydomain.com

My problem is that this new service (GMail) reports that one of the FQDNs hasn't got a valid certificate.
mail.mydomain.com:25 is ok according to Google
out.mydomain.com:25 is not.

What I don't understand is where and how I map the wildcard certificate (visible under /System/Trust) to a IP/FQDN. I am actually surprised that this cert isn't automatically used for all ports and all interfaces hosted on this server?

I searched high and low for settings which would map FQDN or NAT to the existing wildcard cert without success. So my only "logical" explanation is that somehow the certificate is maybe tied to only the first WAN interface mail.*) and not to the virtual IP which out.* points to? But then again, I can't find a way to map the cert to the virtual IP either.

If anyone could provide some ideas as to what and where to look I'd be grateful.

By the way, by joining this forum I also realised that 21.7* is considered legacy, I will have to look into upgrading once this certificate stull is sorted out.