Hey,
running two identical baremetal opnsenses. HA works on LAN as expected. I have an IP range from my provider (Vodafone), but have to route the traffic from the modem to the opnsenses WAN Interfaces via vlan on UniFi switches.
Edit:
If I setup a CARP VIP on WAN Interface the setup seems working. Outbound NAT works too. After a while (some hours) the wan interfaces receive and send a lot of broadcast traffic which knocks out both opnsenses.
If the two WAN Interfaces of the opnsenses are on the same network (same vlan, same IP Range and same subnet mask) a broadcast flood crash the complete network. One OPNsense and an other device (Laptop) is no problem and runs without an issue. Reply-to rule is disabled and both opnseses was rebooted after settings change. If I mark the ports as "isolated network" on the unifi switches (the opnsenses can not see each other) both opnsese are up and running with stable WAN connection, no broadcast flood.
I'm running into the same problem. Same setup as you, IP range from Vodafone on WAN. Using CARP to share 1 public IP between both routers.
Does setting them to "isolated network" mess with the CARP setup? Have you tested failover?
Hi,
the "isolated setting" destroys CARP. So no CARP on WAN is possible but there is also no broadcast flood.
Edit: Problem is the same without CARP configured. 10-15min after pluging in the 2nd opnsense dhcp broadcast flood starts and all switches and opnsenses stopped working. Pull the cable and wait 3 minutes; all runs again.
What I have tried to solve the problem at this point:
- checked if mac adresses of WAN interface are double
- checked no bridges
- disabled reply-to rule
- disabled force-gateway rule
- on vlan site I disabled STP (different versions of STP of your router and Vodafone router = problems)
- checked if it's a hardware failure (used other ports on opnsenses)
-Using Wireshark I couldn't find anything suspicious
...
What drives me crazy is LAN sites (many vlan + normal LAN) runs without issues (CARP + failover). If I plugin the WAN sites of both opnsenses it looks like a layer 2 loop. 1 OPNsense and a third party device (Win10 Laptop + manual static IP) on WAN site no issues .....
I also opened a Reddit post, maybe it will help you: https://www.reddit.com/r/opnsense/comments/1d0g4i9/2_opnsenses_in_same_wansubnet_leads_to_broadcast/
At this point I don't know how to fix the issue .... Any more ideas?
Who is sending DHCP packets into that uplink network? OPNsense? Did you configure WAN as DHCP? Can you just do static instead?
Hi,
WAN ist static public_ip/28 network. The DHCP requests came from both opnsense (source mac adresses), but request IP's for different clients in different vlan. If I plug the "requesting client" out of the swith port, request will come from an other client.
Can provide a wireshark cap if it helps ....
Why are DHCP clients connected to the WAN /28?
Or did I misread your problem entirely? The broadcast storm happens on WAN, right?
Yes broadcast storm happen on WAN. And only if both OPNSenses WAN is plugged in. At the moment both opnsense running, each opnsense connected to different ISP and LAN + local vlan in HA without an issue.
Why this happens, that's the question ....
First bet was misconfigured vlan, but wan vlan is exactly same configuration like lan vlan. Second bet was bridge, but there is no bridge. I use LAGG in my unifi switches ..... but lan ha setup runs without an issue like expected, so I don't think this affects ha setup.
Here some more information:
(public IP censored .....)
root@OPNsense1:~ # ifconfig
ix0: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=48538b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
ether 3c:ec:ef:d9:5d:9a
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: MGMT (lan)
options=48538b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
ether 3c:ec:ef:d9:5d:9b
inet 192.168.90.251 netmask 0xffffff00 broadcast 192.168.90.255
inet 192.168.90.254 netmask 0xffffff00 broadcast 192.168.90.255 vhid 11
inet 192.168.90.1 netmask 0xffffff00 broadcast 192.168.90.255 vhid 6
carp: MASTER vhid 11 advbase 2 advskew 0
carp: MASTER vhid 6 advbase 2 advskew 0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN2_Vodafone (wan)
options=48520b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
ether 3c:ec:ef:d9:58:64
inet ###.###.###.### netmask 0xfffffff0 broadcast ###.###.###.###
groups: WAN
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN1_VSE (opt1)
options=48520b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
ether 3c:ec:ef:d9:58:65
inet 212.82.61.253 netmask 0xffffff00 broadcast 212.82.61.255
groups: WAN
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb2: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=48520b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
ether 3c:ec:ef:d9:58:66
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb3: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=48500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
ether 3c:ec:ef:d9:58:67
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb4: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: RED (opt15)
options=48500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
ether 3c:ec:ef:d9:58:68
inet 10.10.0.251 netmask 0xffffff00 broadcast 10.10.0.255
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb5: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: pfsync (opt13)
options=48500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
ether 3c:ec:ef:d9:58:69
inet 10.0.0.251 netmask 0xffffff00 broadcast 10.0.0.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
enc0: flags=41<UP,RUNNING> metric 0 mtu 1536
groups: enc
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1500
pfsync: syncdev: igb5 syncpeer: 10.0.0.252 maxupd: 128 defer: off
syncok: 1
groups: pfsync
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33160
groups: pflog
ix1_vlan10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: znet (opt3)
options=4000000<NOMAP>
ether 3c:ec:ef:d9:5d:9b
inet 10.10.10.251 netmask 0xffffff00 broadcast 10.10.10.255
inet 10.10.10.1 netmask 0xffffff00 broadcast 10.10.10.255 vhid 9
inet 10.10.10.254 netmask 0xffffff00 broadcast 10.10.10.255 vhid 13
groups: vlan
carp: MASTER vhid 9 advbase 2 advskew 0
carp: MASTER vhid 13 advbase 2 advskew 0
vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VoIP (opt4)
options=4000000<NOMAP>
ether 3c:ec:ef:d9:5d:9b
inet 192.168.20.251 netmask 0xffffff00 broadcast 192.168.20.255
inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255 vhid 8
groups: vlan
carp: MASTER vhid 8 advbase 2 advskew 0
vlan: 20 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: IoT (opt5)
options=4000000<NOMAP>
ether 3c:ec:ef:d9:5d:9b
inet 10.10.30.251 netmask 0xffffff00 broadcast 10.10.30.255
inet 10.10.30.1 netmask 0xffffff00 broadcast 10.10.30.255 vhid 4
groups: vlan
carp: MASTER vhid 4 advbase 2 advskew 0
vlan: 30 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan40: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: Gast (opt6)
options=4000000<NOMAP>
ether 3c:ec:ef:d9:5d:9b
inet 192.168.40.251 netmask 0xffffff00 broadcast 192.168.40.255
inet 192.168.40.1 netmask 0xffffff00 broadcast 192.168.40.255 vhid 12
groups: vlan
carp: MASTER vhid 12 advbase 2 advskew 0
vlan: 40 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan50: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: IPCam (opt7)
options=4000000<NOMAP>
ether 3c:ec:ef:d9:5d:9b
inet 192.168.50.251 netmask 0xffffff00 broadcast 192.168.50.255
inet 192.168.50.1 netmask 0xffffff00 broadcast 192.168.50.255 vhid 5
groups: vlan
carp: MASTER vhid 5 advbase 2 advskew 0
vlan: 50 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: Office (opt2)
options=4000000<NOMAP>
ether 3c:ec:ef:d9:5d:9b
inet 192.168.6.251 netmask 0xffffff00 broadcast 192.168.6.255
inet 192.168.6.1 netmask 0xffffff00 broadcast 192.168.6.255 vhid 1
inet 192.168.6.254 netmask 0xffffff00 broadcast 192.168.6.255 vhid 10
groups: vlan
carp: MASTER vhid 1 advbase 2 advskew 0
carp: MASTER vhid 10 advbase 2 advskew 0
vlan: 6 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan60: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: DMZ (opt8)
options=4000000<NOMAP>
ether 3c:ec:ef:d9:5d:9b
inet 192.168.60.251 netmask 0xffffff00 broadcast 192.168.60.255
inet 192.168.60.1 netmask 0xffffff00 broadcast 192.168.60.255 vhid 2
inet 192.168.60.254 netmask 0xffffff00 broadcast 192.168.60.255 vhid 15
groups: vlan
carp: MASTER vhid 2 advbase 2 advskew 0
carp: MASTER vhid 15 advbase 2 advskew 0
vlan: 60 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan70: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: Tsunami (opt9)
options=4000000<NOMAP>
ether 3c:ec:ef:d9:5d:9b
inet 192.168.70.251 netmask 0xffffff00 broadcast 192.168.70.255
inet 192.168.70.1 netmask 0xffffff00 broadcast 192.168.70.255 vhid 7
groups: vlan
carp: MASTER vhid 7 advbase 2 advskew 0
vlan: 70 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan80: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: Drucker (opt10)
options=4000000<NOMAP>
ether 3c:ec:ef:d9:5d:9b
inet 192.168.80.251 netmask 0xffffff00 broadcast 192.168.80.255
inet 192.168.80.1 netmask 0xffffff00 broadcast 192.168.80.255 vhid 3
groups: vlan
carp: MASTER vhid 3 advbase 2 advskew 0
vlan: 80 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan01: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4000000<NOMAP>
ether 3c:ec:ef:d9:5d:9b
groups: vlan
vlan: 170 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
wg0: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
description: Wireguard (opt14)
options=80000<LINKSTATE>
inet 10.17.66.0 netmask 0xffffff00
groups: wg wireguard
nd6 options=9<PERFORMNUD,IFDISABLED>
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
inet 10.10.17.1 --> 10.10.17.2 netmask 0xffffffff
groups: tun openvpn
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Opened by PID 89184
root@OPNsense1:~ #
dhcprelay running?
No relay is running. DHCP and DHCP6 server is disabled on this Interface.
Then something must be wrong at layer 2 with your switches - the trunks - the VLAN configuration ...
OPNsense will definitely not forward DHCP requests from one interface to another one if there is no DHCP relay.
Well... Thats unfortunate.
In my setup there is MDNS and SSDP traffic leaking from LAN to the WAN site.
No MDNS-repeater setup, no special NAT rules, not even VLANs.
I also tried blocking this traffic with firewall rules, but it seems to only affect about 80% of the packets. Some still get through to the WAN interface and cause a broadcast/multicast loop.
Ok.
In this case, shouldn't there also be a broadcast flood in the LAN or in one of the LAN VLAN? Configuration for WAN vlan is exactly the same like lan vlan configuration. I will delete this evenig the wan vlan and rebuild it from scratch....
To be sure... I checked under Services -> DHCP Relay whether a dhcp relay exists or not.
@HenrikHenkel is IGMP Snooping or Multicast DNS traffic enabled on vlan configuration?
There are no VLANs configured. All dumb switches.
Each network has its own interface in OPNSense.
...you had me at "MDNS"....
You have a Multi WAN Setup?
Is there the possibility that a Firewal rule relay DHCP or DNS traffic to the WAN interface?
EDIT:
Hope this was the solution for me. Testing at the moment ....
EDIT 2:
Sadly not the solution :(. Have a floating rule which allows dhcp,dns,ntp .... to "this firewall" for all local networks. Deactivated the rule, but broadcast flood coming in ...
@aeschma
Yes, I actually do have 2 WAN connections for failover. But those also have their own interfaces.
No firewall rules that should relay this traffic.
The other WAN does not have this problem.
Same Setup here ....
But I can't use my second WAN for HA. Second WAN is DHCP only. So second WAN is configured on both OPNsenses but only plugged into one Sense.... on the other is the interface offline.
Your ISP is also Vodafone? If so, do you think it could be an Vodafone issue?
Yes, my second ISP is Vodafone.
The main WAN is Telekom, no problems there.
Actually it came to my mind, there is a firewall rule that could be the culprit... Because I'm using load-balancing, there's a firewall rule that splits traffic to both WAN interfaces.
I will be on-site on Saturday and will check whether this causes the problem. (Although, if it is... Then it should be on both WAN interfaces, right?)
Do you use load-balancing or just failover?
Yes, I use Load Balancing too.
That's why I asked, but if it was due to Load Balancing, both connections would have to be affected.
I remember a forum post where someone successfully runs HA with Vodafone Cable. So there must exist an solution ....
How did you configure multi WAN? Firewall rule with redirect gateway?
Yes. I have an RFC1918 Alias, which I use to route the public traffic to a Gateway-Group.
So 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, right?
Try to add 224.0.0.0/24, 239.255.0.0/16, 239.192.0.0/14 to the alias.
Ok, I will try it. I can't try it until the weekend because I won't be back before then. I will write you.
I tried it last weekend and it seems like the problem is solved for me.
Kinda feel stupid now, because it should have been obvious from the start to exclude ALL subnets from this firewall rule, that don't belong on a WAN network...
Good to hear it's working for you. Sadly dosen't work for me :(
Here is my alias and firewall rule.
Hi. Sorry for the late response.
The firewall rule and alias look exactly like mine...
Might sound stupid, but did you synchronize the changes to your second firewall?
Yes, both Firewalls are synchonized. I even restarted the firewalls afterwards.
I think it's working now! It's up and running for 8h now.
I switched the STP protocol in the unifi switches from RSTP to STP. Read somewhere that STP is the OPNSense/FreeBSD default (Perhaps the problem is by Vodafone, RSTP + "Local" CARP worked without an issue). In my switch settings was STP deactivated for the WAN port profile .... this is still the case. In tne General Settings on unifi I switched RSTP to STP.
Here are some screenshots of my current setup for people who use OPNSense with unifi switches.
The sad: Now my Multi-WAN connection drops randomly. Internal routing works without an issue but WAN routing drops for a few minutes. But I think this is an other topic ...
Update:
Unfortunately the problem is still there. However, it now occurs much less frequently (approx. once a day) and no longer immediately after plugging in the 2nd firewall.
Did you ever solve this?
Sadly, no :(