Messages

Cuando apago el router principal, el secundario no entra en línea y en VIP->Status aparece:

"CARP has detected a problem and this unit has been demoted to BACKUP status.
Check link status on all interfaces with configured CARP VIPs."
Y el demotion level está en 240.

Las conexiones son correctas.
El puerto del hipervisor en el switch es un trunk con todas las vlans aceptadas.
En ESXI, la máquina virtual tiene configurada una conexión de red a través de un grupo de puertos trunk. Y tiene activado: Modo promiscuo, transmisiones falsificadas, cambios en dirección MAC.

Hola !
Proba con los adaptadores de redes directamente.
cuando vas a firewall / virtual ip / status ...
cuales son los stados de las ip virtuales ?
cual es el valor Current CARP demotion level ?
Saluds

Muchas gracias por tu ayuda.

La idea de usar lag, era evitar la diferencia de nombre del interfaz padre, ya que un dispositivo es físico (nuc) y el otro es virtualizado. De esta forma, en ambos casos se llama "lag".
De todas formas, lo que he hecho es cambiar el tipo de adaptador de la máquina virtual a Intel, y de esa forma ya no se usa lag. Está como dices. Y no funciona.

FW->Virtual IP->Status: Muestra MASTER en el principal y BACKUP en el secundario.
El demotion level, En el principal es 0, y en el secundario es también 0.

Esto justo después de activar el secundario y sincronizar la configuración.
La sincronización funciona y todo parece ir bien, pero unos minutos más tarde 15-30, la red deja de funcionar y la causa parece que sea un caos en el firewall que está bloqueando comunicaciones correctas.

Desde entonces, también probé a hacer un sistema de prueba en el hipervisor, con dos routers virtuales en una red aislada, sin reglas de FW personalizadas, y ocurre exactamente lo mismo.
Llegué a la conclusión que la alta disponibilidad en OPNSense está rota y mantengo el backup apagado.

Hola amigos, espero que me podáis ayudar con la configuración de alta disponibilidad/CARP. Abrí hace ya un tiempo un post en el foro inglés pero no parece que haya mucha intención de ayuda allí.

Estoy intentando poner en marcha un CARP entre un FW físico, en un NUC, y la máquina virtual en ESXI que usaba antes. He conseguido configurar el sistema usando un interface LAGG para abstraer las diferencias del hardware como sugiere la documentación. El NUC solo tiene un interfaz físico y tengo varias Vlans configuradas.

Entiendo que tal y como lo he hecho debería funcionar, ya que si lo comparo con los archivos de ejemplo (a pesar de ser más viejos que la polca...) todo está presente. De hecho PFSync funciona, XMLSync también y los status muestran normal con el fw1 como Master y el fw2 como Backup.
Pero tengo dos problemas gordos:

• 
  El DHCP hace cosas raras, como reservar prácticamente todas las IPs disponibles. Los dispositivos que consiguen IP, obtienen un número del final del rango. (Es normal??). Y si apago el fw2 (Backup), el DHCP deja de funcionar.
  
• El firewall está como loco. El log del Backup muestra bloqueos continuamente, para conexiones para las que tengo reglas de aceptación configuradas. El fw1 de vez en cuando también suelta algunos bloqueos pero en general va bien.
  La comunicación en la red se corta a intervalos regulares. Cada minuto y permanece cortada durante unos cinco minutos. Luego vuelve y así.
  
• Si hago ping al router desde la vlan LAN (mi ordenador) responde al ping el gateway de otra vlan. He comprobado que va cambiando, cada vez que se restaura la conexión, responde desde otra vlan distinta. Rara vez desde la propia.
  

Alguien sabe que pasa aquí? Puede ser que el CARP no funcione en cuanto empiezas a usar vlans y reglas de fw más complejas?
Si alguien entiende, puede pasarle las configuraciones (quitando claves y tal).

[Every minute or so, all communications in the network break, and if left alone, after 2-3mins come back, and then break, etc]

The past 10 days I've been trying to make CARP work without success.

I've got PFSyncs, xml syncs and dhcp somewhat working, but still plenty of issues:

• The DHCP shows almost all the IP leases as reserved. Devices get high number IPs when they get. This is normal? I've tried erasing the dhcp.leases files and reboot in both fw without effect. If backup is offline, DHCP doesn't work -> "dhcpd: DHCPDISCOVER from xxxxx not responding (recovering)"
  
• The firewall goes crazy: After putting both instances online, the backup fw logs full of blocks for traffic that is configured to pass. Main fw seems ok, but from time to time it spits several of those blocks. Every minute or so, all communications in the network break, and if left alone, after 2-3mins come back, and then break, etc... Listening music from my server it's interesting.

The CARP status shows normal.
I've tried to build a configuration from the config examples but it doesn't make a difference.
Is High availability broken?

If someone is willing I could post my configurations, stripping passwords and such.

Hope someone can help me with this.
I had OPNSense running in a VM under esxi for some time. I didn't like loosing the network on server maintenance so I bought  an i3 NUC. While the NUC was coming I modified the existing installation to adapt to the new one.
I had three vlans on a virtual adapter each and the wan in a dedicated passthrough one. I changed it to a single trunk adapter with a router-on-stick configuration, with four vlans.

When I received the NUC I installed OPNS and restored a backup from the VM. Then I thought I could leave the VM and configure a CARP HA.
I followed this: https://www.thomas-krenn.com/en/wiki/OPNsense_HA_Cluster_configuration
and this: https://docs.opnsense.org/manual/how-tos/carp.html

I basically followed those instructions, but created a new vlan interface (also configured on switch) for the PFSync interface.
[As the NUC only has one net adapter (for now) I could not make the "mysterious and undocumented" LAGG overcome to allow syncs, so I left states synchronization deactivated.
I configured XMLRPC Sync.]

I fully configured the LAGG interface and HA settings.

Also, as I have several vlans, when documentation says to create a firewall rule to allow CARP, I created a vlan group with all the intranet+wan vlans and made the rule here*

As I have more than one interface, I created subsequent virtual IPs increasing the VHID group (WAN 1 / LAN 3 / TLN 4 / IOT 5)

The network was operative but I have the following problems:

• 
  When the documentation says to create a fw rule for PFSync interface, "as it is a direct cable"... Mine is not a direct cable, and I didn't know how to create that rule, as documentation also doesn't show it. I created an All to all regular one.
  
• 
  My firewall is full of block impossible hits. Like saying on interface TLN HOST1:80 contacts to HOST2:34234, when host1 is not at TLN but IOT and traffic seems inverted looking at ports. This doesn't affect normal usage.
  
• 
  Failover doesn't work. I shutdown the NUC and network goes over. Not only, but whenever before I could manage switch by setting a fixed ip and attaching my pc on management vlan, now it fails. (could any ARP config in switch mess this?)
  
• 
  Trying to XMLRPC Sync, fails saying that backup is not present. I can ping it and ping the master from the VM. I can see the allowed port 80 connection in the FW2 logs.
  
• 
  The HA->Status menu point also says that there's no communication with the backup node. FW1 seems to hang for a while but FW2 spits the error immediately.
  
• 
  After a day, all the appliances in TLN vlan (trusted) stopped working. The DHCP for this interface was deactivated at configuration (only this one). So I turned it on and started, but it doesn't work and keeps pulling this at log:
  (It does the same in both FWs)
  

Code Select Expand


2020-05-28T12:58:16 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:59 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:51 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:46 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:41 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:36 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:19 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:11 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:07 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:57:02 dhcpd: DHCPDISCOVER from 04:b1:67:1b:d1:62 via em0_vlan10: not responding (recovering)
2020-05-28T12:56:57 dhcpd: failover peer dhcp_lan: I move from startup to communications-interrupted
2020-05-28T12:56:57 dhcpd: failover peer dhcp_opt1: I move from startup to communications-interrupted
2020-05-28T12:56:57 dhcpd: failover peer dhcp_opt2: I move from startup to recover
2020-05-28T12:56:42 dhcpd: Server starting service.
2020-05-28T12:56:42 dhcpd: failover peer dhcp_lan: I move from communications-interrupted to startup
2020-05-28T12:56:42 dhcpd: failover peer dhcp_opt1: I move from communications-interrupted to startup
2020-05-28T12:56:42 dhcpd: failover peer dhcp_opt2: I move from recover to startup
2020-05-28T12:56:42 dhcpd: Sending on   Socket/fallback/fallback-net
2020-05-28T12:56:42 dhcpd: Sending on   BPF/em0_vlan1/f4:4d:30:6a:fb:9c/192.168.0.0/24
2020-05-28T12:56:42 dhcpd: Listening on BPF/em0_vlan1/f4:4d:30:6a:fb:9c/192.168.0.0/24
2020-05-28T12:56:42 dhcpd: Sending on   BPF/em0_vlan50/f4:4d:30:6a:fb:9c/192.168.50.0/24
2020-05-28T12:56:42 dhcpd: Listening on BPF/em0_vlan50/f4:4d:30:6a:fb:9c/192.168.50.0/24
2020-05-28T12:56:42 dhcpd: Sending on   BPF/em0_vlan10/f4:4d:30:6a:fb:9c/192.168.10.0/24
2020-05-28T12:56:42 dhcpd: Listening on BPF/em0_vlan10/f4:4d:30:6a:fb:9c/192.168.10.0/24
2020-05-28T12:56:42 dhcpd: Wrote 150 leases to leases file.
2020-05-28T12:56:42 dhcpd: Wrote 0 new dynamic host decls to leases file.
2020-05-28T12:56:42 dhcpd: Wrote 0 deleted host decls to leases file.
2020-05-28T12:56:42 dhcpd: For info, please visit https://www.isc.org/software/dhcp/
2020-05-28T12:56:42 dhcpd: All rights reserved.
2020-05-28T12:56:42 dhcpd: Copyright 2004-2020 Internet Systems Consortium.
2020-05-28T12:56:42 dhcpd: Internet Systems Consortium DHCP Server 4.4.2
2020-05-28T12:56:42 dhcpd: PID file: /var/run/dhcpd.pid
2020-05-28T12:56:42 dhcpd: Database file: /var/db/dhcpd.leases
2020-05-28T12:56:42 dhcpd: Config file: /etc/dhcpd.conf
2020-05-28T12:56:42 dhcpd: For info, please visit https://www.isc.org/software/dhcp/
2020-05-28T12:56:42 dhcpd: All rights reserved.
2020-05-28T12:56:42 dhcpd: Copyright 2004-2020 Internet Systems Consortium.
2020-05-28T12:56:42 dhcpd: Internet Systems Consortium DHCP Server 4.4.2

Note: Output from before LAGG implementation. Now lagg0_vlan**.

The DHCP in the other two interfaces work as expected. I can see successful DHCP negotiation for other interfaces in the log.

Thank you very much.

EDIT: I can see this block hit on the FW2 when I restart pfsync in the master:
BLOCK   wan      May 29 00:43:12   192.168.8.11   224.0.0.240   pfsync   Block private networks from WAN

This is an automatic rule. I removed it from the interface and added a PFSYNC rule with no effect to HA.

In the attached screenshot, you can observe that there are blocks for icmp for what I have defined a rule in a group that includes this network:

Code Select Expand


Protocol Source Port Destination Port Gateway Schedule Description
  IPv4 ICMP INT net * INT net * * * [INT] Allow ICMP


Now I've deleted the multicast rules, and I can see in the logs that still it's allowing that traffic with a description that says "allow SMS shares on NAS" that it's the following rule after the deleted ones. I suspected configuration corruption, but I've exported and re-imported and makes the same.

That's scary.

In this screenshot, the fw blocking by it's own internet requests of my phone. Before and after multicast rule hits with their correct descriptions.

I should explain what I meant with the point #1 above. If I go to alias, and change the name for an alias, something totally unrelated, even rules in other network stops working. I saw that when I changed the name for a port alias, the igmp rules in IOT network stopped working completely and started to show the blocks in the log. After finding the offending alias (some other where changed but didn't affect) and restoring it's wrong name, the unrelated rules started to work again.
Does that make sense?

Also, for some rules, I deactivated logging, but they still appear in the logs.

Today my opnsense started to act strangely. I have three local interfaces in my system (LAN,TLAN,IOT), with their fw rules and two groups, one that includes all the networks (ALL_LOCAL) and other that only includes user networks (INT).

I have observed the following:

• The rules in my system are not executing in the correct order. And depending on the alias names or group names, they stop working altogether.
• The fw log mixes information of two rules in one line. This is the more apparent and easy to see.
• The fw is blocking (is not processing all the rules until the default block) now and then for internet requests, that are valid.

In the capture you can see that the "allow" hits, that come from my "Allow multicast" rules, show in the description the text for the default WAN block rule. ??
Then in the second capture, by the time I wrote the message, descriptions have changed, but still from other rule. This time a NAT forward one.
Then sometimes, blocks some traffic to internet to some devices, It's normal HTTP/HTTPS traffic. I'll add a screenshot as it happens.
I've rebooted and still does this. What I can do? It is a totally mess.

Thanks!

There must be missing something in the writeup. Implemented it as it states, and lost connectivity. Something should be overlooked regarding firewall rules.

There isn't still any alternative that doesn't require installing external modules and circunventing the system?

[bloqueo de salida]

Pon una regla en el firewall que bloquee paquetes con destino el router 4G, puerto 80 y origen distinto a la subred de la VLAN de confianza

Hola. Te importaría aclararme esto? No es que no entienda lo que dices. Lo que no acabo de entender es como funciona el interfaz del firewall de opnsense. Lo que siempre había hecho, y he trabajado con muchos firewalls, es definir las reglas de bloqueo/protección, en el interfaz que se desea proteger. Es decir, que si no quieres que se acceda a un equipo de la vlan IOT, te ibas a esa red y definías una regla que decia que se bloqueaban las conexiones entrantes a esa IP.
En opnsense me encuentro haciendo las reglas de bloqueo en todos los interfaces excepto en el que se debe proteger, es decir que en lugar de reglas de bloqueo de entrada, son reglas de bloqueo de salida.

No se si me he explicado bien y tiene sentido para tí. Me pregunto si me lo podrías explicar, por favor, por que cada vez que tengo que hacer algo en el sistema, al final me pierdo. Y estoy harto de tener que hacer 25mil reglas cuando solo quiero conseguir una cosa.
Muchas gracias.

That's the version in which I started to get problems.

I just updated from 1.9.2 to 1.9.3 and now it's worse. I've seen internal NFS blockages (I have rules allowing) and one strange line with both external IPs as source and destination.
I'm starting to get scared of this. I think I will activate the silly firewall in the ISP router.

[but my browser never recover: I have to stop the current request, and reload manually]

every time I apply (edit, add, delete, ...) a FW rule, the changes are actually committed, but my browser never recover: I have to stop the current request, and reload manually...

Yes! I also experienced this. Not always, but from time to time.
Look, all the IPs being blocked now are from Google netherlands:
216.58.211.42, 172.217.17.10, 172.217.17.4 All google
161.117.71.92 And this one is Aliexpress cloud. Should be normal as I have a Xiaomi phone and Aliexpress app intalled.

I have deactivated bogon nets detection in case.

PS. I have created a manual default block rule in my wan interface and I'll be monitoring the floating one.
In my case, I have internet connection through a ISP router that doesn't put into bridge, so I have to configure a DMZ and the traffic between the "modem" and opnsense it's using private ips schema. Perhaps that will indicate a pattern.
Now an amazon cdn blocked: 34.216.252.86 -> United States Portland Amazon Technologies Inc.

I'm also interested, as I'm having the same problem, and I would bet that never seen this before the last update.
Did you find the cause for this? I don't think it may be caused by bad hw, as I only have a wan modem and a L2 switch.

Code Select Expand


TLAN Aug 30 16:18:43 192.168.10.55:46524 74.125.133.92:443 tcp Default deny rule
TLAN Aug 30 16:18:43 192.168.10.55:46524 74.125.133.92:443 tcp Default deny rule
TLAN Aug 30 16:18:43 192.168.10.55:46524 74.125.133.92:443 tcp Default deny rule
TLAN Aug 30 16:18:43 192.168.10.55:46524 74.125.133.92:443 tcp Default deny rule
TLAN Aug 30 16:18:43 192.168.10.55:40513 216.58.211.42:443 tcp Default deny rule
TLAN Aug 30 16:18:43 192.168.10.55:44061 31.13.83.51:443 tcp Default deny rule
TLAN Aug 30 16:18:42 192.168.10.55:44061 31.13.83.51:443 tcp Default deny rule
TLAN Aug 30 16:18:42 192.168.10.55:46011 172.217.17.10:443 tcp Default deny rule
TLAN Aug 30 16:18:41 192.168.10.55:40513 216.58.211.42:443 tcp Default deny rule
TLAN Aug 30 16:18:41 192.168.10.55:46011 172.217.17.10:443 tcp Default deny rule
TLAN Aug 30 16:18:41 192.168.10.55:46011 172.217.17.10:443 tcp Default deny rule
TLAN Aug 30 16:18:41 192.168.10.55:40513 216.58.211.42:443 tcp Default deny rule


For the nonce, it seems to affect wifi devices accesing internet (google cdns I would say) because I've seen my phone, wifes and the TV (android)

TLAN is my trusted lan and it has no other rules than default. Also, I don't know if it's normal that the floating rules seem to have order inverted, so "default deny rule" sits the first of them.

@mapsware Aún no las tengo todas conmigo. Hace cosas muy raras.
Cómo por ejemplo, que desde la vlan de confianza (como interface en opn) sea capaz de entrar en el portal de administración del router 4g directamente. Solo desde este interfaz. Cuando no debería ser posible acceder a la IP del gateway de la wan.

Enviado desde mi MI 5s Plus mediante Tapatalk