[SOLVED] 19.7.4: Rebooting the switch - OPNsense looses WAN connection/routing

[rainerle]

Hi,

after rebooting the switch attached to the MASTER OPNsense the firewall loses its WAN connection/routing, but does not failover to the BACKUP.

Only after executing menu option 11 (Reload all services) WAN is working again.

Logfile

Code: [Select]

Sep 30 23:11:44 opnsense01 sshd[47244]: Connection closed by 10.20.30.28 port 39584 [preauth]
Sep 30 23:12:44 opnsense01 sshd[12208]: Connection closed by 10.20.30.28 port 41114 [preauth]
Sep 30 23:13:44 opnsense01 sshd[38112]: Connection closed by 10.20.30.28 port 42595 [preauth]
Sep 30 23:14:44 opnsense01 sshd[60325]: Connection closed by 10.20.30.28 port 44062 [preauth]
Sep 30 23:15:04 opnsense01 kernel: ixl0: link state changed to DOWN
Sep 30 23:15:04 opnsense01 kernel: ixl1: link state changed to DOWN
Sep 30 23:15:04 opnsense01 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for admin_port(lan) but ignoring since interface is configured with static IP (10.11.10.11 ::)
Sep 30 23:15:06 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan3)
Sep 30 23:15:06 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2026)
Sep 30 23:15:06 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2040)
Sep 30 23:15:06 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan30)
Sep 30 23:15:06 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2038)
Sep 30 23:15:06 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2028)
Sep 30 23:15:06 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2027)
Sep 30 23:15:06 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2020)
Sep 30 23:15:26 opnsense01 kernel: pflog0: promiscuous mode disabled
Sep 30 23:15:26 opnsense01 kernel: pflog0: promiscuous mode enabled
Sep 30 23:15:44 opnsense01 sshd[94285]: Connection closed by 10.20.30.28 port 45496 [preauth]
Sep 30 23:16:22 opnsense01 kernel: pflog0: promiscuous mode disabled
Sep 30 23:16:22 opnsense01 kernel: pflog0: promiscuous mode enabled
Sep 30 23:16:44 opnsense01 sshd[5997]: Connection closed by 10.20.30.28 port 46934 [preauth]
Sep 30 23:17:44 opnsense01 sshd[12793]: Connection closed by 10.20.30.28 port 48371 [preauth]
Sep 30 23:18:44 opnsense01 kernel: ixl0: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Sep 30 23:18:44 opnsense01 kernel: ixl0: link state changed to UP
Sep 30 23:18:44 opnsense01 kernel: ixl1: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Sep 30 23:18:44 opnsense01 kernel: ixl1: link state changed to UP
Sep 30 23:18:44 opnsense01 sshd[76502]: Connection closed by 10.20.30.28 port 49844 [preauth]
Sep 30 23:18:44 opnsense01 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for admin_port(lan) but ignoring since interface is configured with static IP (10.11.10.11 ::)
Sep 30 23:18:45 opnsense01 opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'ixl0'
Sep 30 23:18:45 opnsense01 opnsense: /usr/local/etc/rc.newwanip: On (IP address: 10.11.10.11) (interface: admin_port[lan]) (real interface: ixl0).
Sep 30 23:18:45 opnsense01 opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'lan'
Sep 30 23:18:45 opnsense01 opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan
Sep 30 23:18:45 opnsense01 opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv4 default route
Sep 30 23:18:45 opnsense01 opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to wan
Sep 30 23:18:45 opnsense01 opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
Sep 30 23:18:45 opnsense01 opnsense: /usr/local/etc/rc.newwanip: Removing static route for monitor 2001:4860:4860::8888 via <CARP WAN IPv6>
Sep 30 23:18:45 opnsense01 opnsense: /usr/local/etc/rc.newwanip: Adding static route for monitor 2001:4860:4860::8888 via <CARP WAN IPv6>
Sep 30 23:18:45 opnsense01 opnsense: /usr/local/etc/rc.newwanip: Removing static route for monitor 8.8.8.8 via <CARP WAN IP>
Sep 30 23:18:45 opnsense01 opnsense: /usr/local/etc/rc.newwanip: Adding static route for monitor 8.8.8.8 via <CARP WAN IP>
Sep 30 23:18:50 opnsense01 kernel: pflog0: promiscuous mode disabled
Sep 30 23:18:50 opnsense01 opnsense: /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface admin_port.
Sep 30 23:18:50 opnsense01 kernel: pflog0: promiscuous mode enabled
Sep 30 23:18:57 opnsense01 kernel: pflog0: promiscuous mode disabled
Sep 30 23:18:57 opnsense01 kernel: pflog0: promiscuous mode enabled
Sep 30 23:19:02 opnsense01 kernel: pflog0: promiscuous mode disabled
Sep 30 23:19:02 opnsense01 kernel: pflog0: promiscuous mode enabled
Sep 30 23:19:07 opnsense01 kernel: pflog0: promiscuous mode disabled
Sep 30 23:19:07 opnsense01 kernel: pflog0: promiscuous mode enabled
Sep 30 23:19:24 opnsense01 kernel: pflog0: promiscuous mode disabled

ixl0 is a access port attached on that rebooted switch, ixl1 is the LACP lagg port attached to that same switch. The second switch stays alive and takes over operation during the reboot of switch 1. WAN connection fails as soon as the rebooted switch comes up again.

No idea how to go from here...

[rainerle]

So taking the Ports down on the firewall down one by one -> Logfile:

Code: [Select]

Oct  1 00:13:44 opnsense01 sshd[232]: Connection closed by 10.20.30.28 port 44770 [preauth]
Oct  1 00:14:44 opnsense01 sshd[15068]: Connection closed by 10.20.30.28 port 46214 [preauth]
Oct  1 00:15:44 opnsense01 sshd[37649]: Connection closed by 10.20.30.28 port 47687 [preauth]
Oct  1 00:16:27 opnsense01 kernel: ixl0: link state changed to DOWN
Oct  1 00:16:28 opnsense01 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for admin_port(lan) but ignoring since interface is configured with static IP (10.11.10.11 ::)
Oct  1 00:16:44 opnsense01 sshd[58393]: Connection closed by 10.20.30.28 port 49155 [preauth]
 Oct  1 00:17:17 opnsense01 kernel: ixl0: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Oct  1 00:17:17 opnsense01 kernel: ixl0: link state changed to UP
Oct  1 00:17:18 opnsense01 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for admin_port(lan) but ignoring since interface is configured with static IP (10.11.10.11 ::)
Oct  1 00:17:18 opnsense01 opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'ixl0'
Oct  1 00:17:18 opnsense01 opnsense: /usr/local/etc/rc.newwanip: On (IP address: 10.11.10.11) (interface: admin_port[lan]) (real interface: ixl0).
Oct  1 00:17:44 opnsense01 sshd[17707]: Connection closed by 10.20.30.28 port 50639 [preauth]
Oct  1 00:18:44 opnsense01 sshd[72812]: Connection closed by 10.20.30.28 port 52126 [preauth]
Oct  1 00:18:49 opnsense01 kernel: ixl1: link state changed to DOWN
Oct  1 00:18:52 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2028)
Oct  1 00:18:52 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan3)
Oct  1 00:18:52 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2040)
Oct  1 00:18:52 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan30)
Oct  1 00:18:52 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2020)
Oct  1 00:18:52 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2038)
Oct  1 00:18:52 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2026)
Oct  1 00:18:52 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2027)
Oct  1 00:19:14 opnsense01 kernel: ixl1: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Oct  1 00:19:14 opnsense01 kernel: ixl1: link state changed to UP
Oct  1 00:19:44 opnsense01 sshd[46354]: Connection closed by 10.20.30.28 port 53587 [preauth]

All good, no problems with the WAN connection on the firewall. Must be something else, maybe the length of the downtime or both interfaces coming down and up at the same time.

[rainerle]

To prepare for suricata I changed to configuration of the network interfaces:

• Got rid of LACP lagg devices
• ixl0 - LAN - admin_port
• ixl1 - WAN - uplink interface (the switchport is assigned to the proper VLAN)
• ixl2 - VLANs for all other OPT interfaces
• ixl3 - pfSync interface directly attached to the backup OPNsense

Since getting rid of the LACP lagg devices the reboot of the switch attached to the master OPNsense does work nicely: The CARPs fail over to the backup OPNsense, as soon as the master's switch comes up again the master OPNsense takes over.