Topics

In the past, I had CARP active with approx. 20-25 VLANs. As I had stability problems, I implemented the second firewall as a cold standby at some point and deactivated CARP.
Now I'm wondering what I did wrong a few years ago and am asking myself the following question:
If one of the VLAN changes the master status to slave, shouldn't all the other VLANs automatically change to slave as well, because otherwise I have a 'split brain problem'. This has never happened to me. It happened that one or more VLANs had the status slave, while the rest were still master.
What do you think?
Thank You!

I have many VLANs where the classic ISC DHCP SErver does its job. Now I wanted to deactivate ISC and activate KEA in one VLAN as a test, but I can't get a DHCP lease. I don't see any errors in the log file and the KEA service starts. Do I have to disable ISC on all interfaces for it to work?

I have already configured the corresponding subnet in KEA and checked it three times. But after deactivating ISC and activating KEA on that specific VLAN, my client does not get a DHCP lease on the VLAN.

Thank you!

Hello,
A colleague installed the update from 24.7 to 24.7.1 during the night. It was this morning when I noticed that the OpenVPN connections can no longer be established. All certificates report "error=cerificate revoked".
We couldn't find any indication of why this was suddenly the case, so I set "Peer Certificate Revocation List = none" in the server instance. Since then the login has been working again.
We are still using the legacy OpenVPN server instances. Did I miss something critical in the changelog?

Code Select Expand

2024-08-16T07:24:54 Error openvpn_server8 <ip-address>:3733 TLS_ERROR: BIO read tls_read_plaintext error
2024-08-16T07:24:54 Error openvpn_server8 <ip-address>:3733 OpenSSL: error:0A000086:SSL routines::certificate verify failed:
2024-08-16T07:24:54 Error openvpn_server8 <ip-address>:3733 VERIFY ERROR: depth=1, error=certificate revoked: C=DE, ST=<city>, L=<city>, O=<org>, emailAddress=<e-mail>, CN=gateway-internal-ca, serial=0
2024-08-16T07:24:54 Error openvpn_server8 <ip-address>:3728 Fatal TLS error (check_tls_errors_co), restarting
2024-08-16T07:24:54 Error openvpn_server8 <ip-address>:3728 TLS Error: TLS handshake failed
2024-08-16T07:24:54 Error openvpn_server8 <ip-address>:3728 TLS Error: TLS object -> incoming plaintext read error
2024-08-16T07:24:54 Error openvpn_server8 <ip-address>:3728 TLS_ERROR: BIO read tls_read_plaintext error
2024-08-16T07:24:54 Error openvpn_server8 <ip-address>:3728 OpenSSL: error:0A000086:SSL routines::certificate verify failed:
2024-08-16T07:24:54 Error openvpn_server8 <ip-address>:3728 VERIFY ERROR: depth=1, error=certificate revoked: C=DE, ST=<city>, L=<city>, O=<org>, emailAddress=<e-mail>, CN=gateway-internal-ca, serial=0
2024-08-16T07:24:54 Error openvpn_server8 <ip-address>41728 Fatal TLS error (check_tls_errors_co), restarting
2024-08-16T07:24:54 Error openvpn_server8 <ip-address>41728 TLS Error: TLS handshake failed
2024-08-16T07:24:54 Error openvpn_server8 <ip-address>41728 TLS Error: TLS object -> incoming plaintext read error
2024-08-16T07:24:54 Error openvpn_server8 <ip-address>41728 TLS_ERROR: BIO read tls_read_plaintext error
2024-08-16T07:24:54 Error openvpn_server8 <ip-address>41728 OpenSSL: error:0A000086:SSL routines::certificate verify failed:
2024-08-16T07:24:54 Error openvpn_server8 <ip-address>41728 VERIFY ERROR: depth=1, error=certificate revoked: C=DE, ST=<city>, L=<city>, O=<org>, emailAddress=<e-mail>, CN=gateway-internal-ca, serial=0
2024-08-16T07:24:53 Error openvpn_server8 <ip-address>:3737 Fatal TLS error (check_tls_errors_co), restarting
2024-08-16T07:24:53 Error openvpn_server8 <ip-address>:3737 TLS Error: TLS handshake failed
2024-08-16T07:24:53 Error openvpn_server8 <ip-address>:3737 TLS Error: TLS object -> incoming plaintext read error
2024-08-16T07:24:53 Error openvpn_server8 <ip-address>:3737 TLS_ERROR: BIO read tls_read_plaintext error
2024-08-16T07:24:53 Error openvpn_server8 <ip-address>:3737 OpenSSL: error:0A000086:SSL routines::certificate verify failed:

Hi!

I have several free SFP+ ports on my hardware. My goal is to secure my public IPs over two lines to the ISP Switch. However, an IP alias can only be assigned to one interface. How could I achieve this goal without using two Firewalls with CARP?
Thank you :-)

Please help me to understand the live log correctly. I am having extreme problems looking up anything there for debugging purposes.  I mostly switch back to TCPDUMP.

For example I have a mailserver. I go into the livelog, enter the filter "src IP mailserver" or "dst IP mailserver" and nothing is displayed.
Neither my ping tests, nor the mails coming in, nor anything else.

At the same time, several mails per second go through this server, but I can't see anything in the livelog.

Do I need to check "enable logging" everywhere in the ruleset?

The mail server is in a DMZ, which means that all connections have to go through the firewall and therefore have to be visible in the livelog.

Maybe I have too many connections, so it can't show up in the livelog? I do not know exactly where I see the active connections, but the firewall has 32k active states.

don't get me wrong, when i run the livelog without filter, i see dozens of entries that change very quickly.

Hallo,
ich bin gerade am grübeln, wie man bei folgendem Szenario richtig vorgeht.

Angenommen ich habe dutzende VLAN's und möchte nun eine zweite Firewall zwecks HA einbinden.
Muss ich denn jetzt auch für jedes VLAN eine separate CARP Adresse konfigurieren, oder genügt es, wenn ich auf einem Interface CARP konfiguriere, damit die Firewalls den Master/Slave aushandeln können, und für alle weiteren VLAN/Interfaces richte ich dann Virtuelle IP's ein?

In meinem Fall habe ich an die 50 VLAN's und kann mich ehrlich gesagt nicht damit anfreunden für jedes Interface eine extra CARP IP einzurichten.

Danke!

Hallo Leute,
seit einiger Zeit übertrage ich die Logfiles schon an einen Loki Server. Nun will ich gerade ein bestimmtes Problem debuggen, wo die Fehlermeldungen hier auflaufen:

    System: Log Files: General

Aber jene Meldungen kann ich nicht via Loki finden, sämtliche anderen jedoch schon.
Ich habe nochmal kontrolliert und bei Logging destination sind alle Applications angehakt, außer filter (filterlog). Hat da jemand eine Idee zu?

Danke!

Hello,
I've been wondering for a while if I've been using CARP incorrectly for years and if I can't do better.

I have a lot of VLANs, currently around 80-100 I guess, mostly /29 networks for customer environments for security purposes.
Now I have also configured a CARP address for each VLAN, but is that really necessary?

Isn't it enough if I set up CARP only in the main network, for example, and set up a Virtual IP for all other interfaces/VLANs? As soon as a problem is detected in the main network, the master moves to the slave and with it all virtual IPs.

How would you do it?
Thank you!

I have a strange problem since Sunday.
Two hardware identical firewalls are working in HA mode with CARP. OPNsense 22.1.6. It works for years.

Since Sunday I have packet loss on all VLAN's. Both firewalls have problems to define the CARP master. As a first step I shut down the slave. The log then throws the same message across all CARP interfaces, the order is random (master firewall):

example:
carp: 60@ixl1_vlan141: BACKUP-> MASTER (master timed out)
carp: 60@ixl1_vlan141: MASTER -> INIT (hardware interface up)
carp: 60@ixl1_vlan141: INIT -> BACKUP (initialization complete)
carp: 60@ixi1_vlan141: BACKUP -> MASTER (master timed out)

and again: it is configured as master itself and its slave is offline. Base 3 and skew 0 is configured.

Next I updated the master to 22.7.2, because of a possible software bug. No success.
Then I replaced the Intel card with Mellanox. No success.
With the current version 22.7.2 the DHCP service also fails at some point and does not assign any more leases until the reboot.

Then I changed a VLAN CARP IP to the IP alias to exclude CARP - drops still follow.

Packet losses occur exclusively to the firewall, or across the VLANs. VLAN internally from host to host there are no losses.
It does'nt matter if the Master is the only one online, or the Slave, or both.
Does anyone have an idea what else I can do?

Hi,
we have not too long ago installed Zenarmor in the community edition for testing. We have left everything mostly on default settings and only the "Block Malware Activity" filter active.

Now we have noticed that sporadic network traffic between VLANs does not work properly. After a while it turned out that it was Zenarmor that was blocking the traffic. However, it was not directly obvious to us. I had looked under reports for threats and blocks, but did not see anything suspicious.

Long story short: is there a livelog where you can see blocking states directly?

Thank You!

Hello, I have recently the problem that my master without warning becomes the backup server, and the slave to the master. I still can't find a clue why this happens, but in the logfile for each interface is the following entry:

2022-04-27T16:06:18 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "VLAN146 (10.27.146.250) (89@ixl1_vlan146)" has resumed the state "BACKUP" for vhid 89

I can't find any other hint in the log indicating any error.

The only difference between master and slave is that the master is running a zenarmor. can this be the cause?

EDIT: is there a way to configure a threshold like lets say "wait 5 seconds until change master/slave"?

Hi,
I'm testing Zenarmor since a couple of Days (free version) and as I logged into my Opnsense Dashboard this morning I saw, that the 75% of my RAM was used, , still growing (25Gig of 32Gig).

My first task was to restart Zenarmor and it freed ~10Gig. I'm Using the latest Version of Zenarmor and it is configured to act in Passive Mode (Reporting Only). What could be a reason?

Zenarmors statistics says, that I have 1200 unique local devices (don't realy believe that). Is my Memory not enough for that?

Are there any other things that I can check?

Thank you!

Hi,
i updated today from latest 21.x to 22.1.2_1 and i'm facing a problem now, that 3 of ~13 IPSec tunnels are not correctly online. They are connected but i dont get traffic through it.

The Logile does not work any more, even for openvpn. "No results found!". so i have trouble to debug this.

does anyone know something about that?

sidenote: the new ipsec overview with the P1+P2.. mh i dont really like it :)

Thank you!

edit: ovpn logfile is working again, but not for ipsec
edit2: found out that i have to change the loglevel view..
edit3:

there is nothing special to find in the log, everything seems to be okay, but it is not

Hello,
I do not understand the live log. I see all sorts of information there, but when I specifically want to investigate a case for which there is a firewall rule, I don't see it.

Inside the rule there is a check mark "Log packets that are handled by this rule". For example, if I ping this server that is noted in the rule, it doesn't show up in the livelog. Why not?

I can put in a simple rule "icmp allow to server IP 1.2.3.4", I set the checkbox and look in the live log if it is being tracked. I start my ping (works) and watch the log but no entry to be found.

I keep running into this problem so the live log has never given me any benefit to date. I have then always used tcpdump on the console.

Am I missing something fundamental? It also doesn't matter which filter I use, i.e. whether I search for source , destination or "address", I don't see what I need.

Thank you for your time :)

Edit: is there an option to log every thing? I mean my log destination is a RAM disk with 30Gigs free space. Time Range? usually never needed the last past hours, only the time "now"

Hi!
I have many VLANs and most of them only need a default ruleset like "allow DNS, forbid private networks, allow internet". I thought that you can pretty well slay with firewall groups and not create the same rule sets over and over again. But what do I do if one of these VLANs needs an additional rule? Do I have to take it out of the firewall group and build individual rules again?

Thanks!

Hello,
I have a question about the firewall Livelog. Should this really show everything?
I am debugging an OpenVPN Site2Site issue, but I don't see the connection attempts in the live log. Not even when the filter is set to "address", "Src", or "Dst". In the OpenVPN log I see all connection attempts, but why not in the firewall live log?

Thanks!

Hi all,
I am planning a new HA setup for a branch. Is it possible to implement the whole thing with different hardware, specifically I imagine that server 1 gets a Mellanox card and server 2 an Intel. As far as I can remember, only the order and description of the interfaces must remain the same, or am I wrong?

The goal is that I want to spread the "risk" as far as possible, if there is a firmware bug or something similar, not both systems are affected.

Thank you!

[ON VLANS]

Hello, I've been monitoring my network traffic for a while with IDS and would now like to use IPS ON VLANS.

I have a Mellanox Connect 4 installed (I think  ::)) and get the following error message when starting with IPS:

This is my used driver.

dev.mlx4_core.0.%pnpinfo: vendor=0x15b3 device=0x1007 subvendor=0x103c subdevice=0x801f class=0x020000
dev.mlx4_core.0.%location: slot=0 function=0 dbsf=pci0:1:0:0 handle=\_SB_.PCI0.PEG0.PEGP
dev.mlx4_core.0.%driver: mlx4_core
dev.mlx4_core.0.%desc: Mellanox driver (3.5.1)

All hardware offload settings are disabled. Can I actively do something, or is it a bug within the last realeses with 21.7.7?

This is the failure log notice:

Code Select Expand

2021-12-19T06:48:30 suricata[73213] [100528] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - opening devname netmap:mlxen0_vlan949/R failed: Invalid argument

thanks a lot!

Hello,
I have many VLANs on my firewalls, now I wanted to add more and then I run into a problem on the "Virtual IP" step.

On the master the Virtual IP has this value, which was also given by me:
VLAN137@79 with the IP 10.27.137.250.

Now I start the sync on my slave. By the way, there the interfaces are named exactly the same as on the master.
But there the following is set up:

VLAN138@79 with IP 10.27.137.250

So the sync transmits the wrong VLAN interface to the slave. But why?

I had this error several times and could help myself so far by deleting the VLAN interface on both sides and creating a new one. Only deleting the virtual IP, synchronize and create again was never enough.

But at the moment I have to create a lot of VLANs and then this becomes a very annoying matter. Maybe someone knows where I can look for the error here?

EDIT:
if I manually correct the VLAN interface on the slave, CARP works as it should, but the next sync overwrites it again


Thanks a lot!

Hi,
Yesterday I installed a brand new server with OPNSense on ZFS basis. Today I had to reboot to check things in the BIOS. Then I noticed that the server no longer boots into the text console, but something graphical. The server itself is normally accessible via WebUI. There is no boot stick or virtual media in it. I have also tried with Linux Style Alt+F1 - F12 to change the console, but had no success. Can I edit a file somewhere via SSH that restores the classic boot? Have a look at the screenshot

Any ideas?
Thanks!