2 OPNsenses same WAN network Broadcast Flood

[aeschma]

Hey,
running two identical baremetal opnsenses. HA works on LAN as expected. I have an IP range from my provider (Vodafone), but have to route the traffic from the modem to the opnsenses WAN Interfaces via vlan on UniFi switches.


Edit:

If I setup a CARP VIP on WAN Interface the setup seems working. Outbound NAT works too. After a while (some hours) the wan interfaces receive and send a lot of broadcast traffic which knocks out both opnsenses.

If the two WAN Interfaces of the opnsenses are on the same network (same vlan, same IP Range and same subnet mask) a broadcast flood crash the complete network. One OPNsense and an other device (Laptop) is no problem and runs without an issue. Reply-to rule is disabled and both opnseses was rebooted after settings change. If I mark the ports as "isolated network" on the unifi switches (the opnsenses can not see each other) both opnsese are up and running with stable WAN connection, no broadcast flood.

[HenrikHenkel]

I'm running into the same problem. Same setup as you, IP range from Vodafone on WAN. Using CARP to share 1 public IP between both routers.

Does setting them to "isolated network" mess with the CARP setup? Have you tested failover?

[aeschma]

Hi,

the "isolated setting" destroys CARP. So no CARP on WAN is possible but there is also no broadcast flood.

Edit: Problem is the same without CARP configured. 10-15min after pluging in the 2nd opnsense dhcp broadcast flood starts and all switches and opnsenses stopped working. Pull the cable and wait 3 minutes; all runs again.

What I have tried to solve the problem at this point:

- checked if mac adresses of WAN interface are double
- checked no bridges
- disabled reply-to rule
- disabled force-gateway rule
- on vlan site I disabled STP (different versions of STP of your router and Vodafone router = problems)
- checked if it's a hardware failure (used other ports on opnsenses)
-Using Wireshark I couldn't find anything suspicious
...


What drives me crazy is LAN sites (many vlan + normal LAN) runs without issues (CARP + failover). If I plugin the WAN sites of both opnsenses it looks like a layer 2 loop. 1 OPNsense and a third party device (Win10 Laptop + manual static IP) on WAN site no issues .....


I also opened a Reddit post, maybe it will help you: https://www.reddit.com/r/opnsense/comments/1d0g4i9/2_opnsenses_in_same_wansubnet_leads_to_broadcast/

At this point I don't know how to fix the issue .... Any more ideas?

[Patrick M. Hausen]

Who is sending DHCP packets into that uplink network? OPNsense? Did you configure WAN as DHCP? Can you just do static instead?

[aeschma]

Hi,

WAN ist static public_ip/28 network. The DHCP requests came from both opnsense (source mac adresses), but request IP's for different clients in different vlan. If I plug the "requesting client" out of the swith port, request will come from an other client.

Can provide a wireshark cap if it helps ....

[Patrick M. Hausen]

Why are DHCP clients connected to the WAN /28?

Or did I misread your problem entirely? The broadcast storm happens on WAN, right?

[aeschma]

Yes broadcast storm happen on WAN. And only if both OPNSenses WAN is plugged in. At the moment both opnsense running, each opnsense connected to different ISP and LAN + local vlan in HA without an issue.

Why this happens, that's the question ....

First bet was misconfigured vlan, but wan vlan is exactly same configuration like lan vlan.  Second bet was bridge, but there is no bridge. I use LAGG in my unifi switches ..... but lan ha setup runs without an issue like expected, so I don't think this affects ha setup.

[aeschma]

Here some more information:
(public IP censored .....)

Code: [Select]

root@OPNsense1:~ # ifconfig
ix0: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=48538b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
        ether 3c:ec:ef:d9:5d:9a
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: MGMT (lan)
        options=48538b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
        ether 3c:ec:ef:d9:5d:9b
        inet 192.168.90.251 netmask 0xffffff00 broadcast 192.168.90.255
        inet 192.168.90.254 netmask 0xffffff00 broadcast 192.168.90.255 vhid 11
        inet 192.168.90.1 netmask 0xffffff00 broadcast 192.168.90.255 vhid 6
        carp: MASTER vhid 11 advbase 2 advskew 0
        carp: MASTER vhid 6 advbase 2 advskew 0
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN2_Vodafone (wan)
        options=48520b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
        ether 3c:ec:ef:d9:58:64
        inet ###.###.###.### netmask 0xfffffff0 broadcast ###.###.###.###
        groups: WAN
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN1_VSE (opt1)
        options=48520b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
        ether 3c:ec:ef:d9:58:65
        inet 212.82.61.253 netmask 0xffffff00 broadcast 212.82.61.255
        groups: WAN
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb2: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=48520b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
        ether 3c:ec:ef:d9:58:66
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb3: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=48500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
        ether 3c:ec:ef:d9:58:67
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb4: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: RED (opt15)
        options=48500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
        ether 3c:ec:ef:d9:58:68
        inet 10.10.0.251 netmask 0xffffff00 broadcast 10.10.0.255
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb5: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: pfsync (opt13)
        options=48500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
        ether 3c:ec:ef:d9:58:69
        inet 10.0.0.251 netmask 0xffffff00 broadcast 10.0.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
enc0: flags=41<UP,RUNNING> metric 0 mtu 1536
        groups: enc
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1500
        pfsync: syncdev: igb5 syncpeer: 10.0.0.252 maxupd: 128 defer: off
        syncok: 1
        groups: pfsync
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33160
        groups: pflog
ix1_vlan10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: znet (opt3)
        options=4000000<NOMAP>
        ether 3c:ec:ef:d9:5d:9b
        inet 10.10.10.251 netmask 0xffffff00 broadcast 10.10.10.255
        inet 10.10.10.1 netmask 0xffffff00 broadcast 10.10.10.255 vhid 9
        inet 10.10.10.254 netmask 0xffffff00 broadcast 10.10.10.255 vhid 13
        groups: vlan
        carp: MASTER vhid 9 advbase 2 advskew 0
        carp: MASTER vhid 13 advbase 2 advskew 0
        vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: VoIP (opt4)
        options=4000000<NOMAP>
        ether 3c:ec:ef:d9:5d:9b
        inet 192.168.20.251 netmask 0xffffff00 broadcast 192.168.20.255
        inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255 vhid 8
        groups: vlan
        carp: MASTER vhid 8 advbase 2 advskew 0
        vlan: 20 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: IoT (opt5)
        options=4000000<NOMAP>
        ether 3c:ec:ef:d9:5d:9b
        inet 10.10.30.251 netmask 0xffffff00 broadcast 10.10.30.255
        inet 10.10.30.1 netmask 0xffffff00 broadcast 10.10.30.255 vhid 4
        groups: vlan
        carp: MASTER vhid 4 advbase 2 advskew 0
        vlan: 30 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan40: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Gast (opt6)
        options=4000000<NOMAP>
        ether 3c:ec:ef:d9:5d:9b
        inet 192.168.40.251 netmask 0xffffff00 broadcast 192.168.40.255
        inet 192.168.40.1 netmask 0xffffff00 broadcast 192.168.40.255 vhid 12
        groups: vlan
        carp: MASTER vhid 12 advbase 2 advskew 0
        vlan: 40 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan50: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: IPCam (opt7)
        options=4000000<NOMAP>
        ether 3c:ec:ef:d9:5d:9b
        inet 192.168.50.251 netmask 0xffffff00 broadcast 192.168.50.255
        inet 192.168.50.1 netmask 0xffffff00 broadcast 192.168.50.255 vhid 5
        groups: vlan
        carp: MASTER vhid 5 advbase 2 advskew 0
        vlan: 50 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Office (opt2)
        options=4000000<NOMAP>
        ether 3c:ec:ef:d9:5d:9b
        inet 192.168.6.251 netmask 0xffffff00 broadcast 192.168.6.255
        inet 192.168.6.1 netmask 0xffffff00 broadcast 192.168.6.255 vhid 1
        inet 192.168.6.254 netmask 0xffffff00 broadcast 192.168.6.255 vhid 10
        groups: vlan
        carp: MASTER vhid 1 advbase 2 advskew 0
        carp: MASTER vhid 10 advbase 2 advskew 0
        vlan: 6 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan60: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: DMZ (opt8)
        options=4000000<NOMAP>
        ether 3c:ec:ef:d9:5d:9b
        inet 192.168.60.251 netmask 0xffffff00 broadcast 192.168.60.255
        inet 192.168.60.1 netmask 0xffffff00 broadcast 192.168.60.255 vhid 2
        inet 192.168.60.254 netmask 0xffffff00 broadcast 192.168.60.255 vhid 15
        groups: vlan
        carp: MASTER vhid 2 advbase 2 advskew 0
        carp: MASTER vhid 15 advbase 2 advskew 0
        vlan: 60 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan70: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Tsunami (opt9)
        options=4000000<NOMAP>
        ether 3c:ec:ef:d9:5d:9b
        inet 192.168.70.251 netmask 0xffffff00 broadcast 192.168.70.255
        inet 192.168.70.1 netmask 0xffffff00 broadcast 192.168.70.255 vhid 7
        groups: vlan
        carp: MASTER vhid 7 advbase 2 advskew 0
        vlan: 70 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1_vlan80: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Drucker (opt10)
        options=4000000<NOMAP>
        ether 3c:ec:ef:d9:5d:9b
        inet 192.168.80.251 netmask 0xffffff00 broadcast 192.168.80.255
        inet 192.168.80.1 netmask 0xffffff00 broadcast 192.168.80.255 vhid 3
        groups: vlan
        carp: MASTER vhid 3 advbase 2 advskew 0
        vlan: 80 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan01: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4000000<NOMAP>
        ether 3c:ec:ef:d9:5d:9b
        groups: vlan
        vlan: 170 vlanproto: 802.1q vlanpcp: 0 parent interface: ix1
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
wg0: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
        description: Wireguard (opt14)
        options=80000<LINKSTATE>
        inet 10.17.66.0 netmask 0xffffff00
        groups: wg wireguard
        nd6 options=9<PERFORMNUD,IFDISABLED>
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet 10.10.17.1 --> 10.10.17.2 netmask 0xffffffff
        groups: tun openvpn
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 89184
root@OPNsense1:~ #


[Patrick M. Hausen]

dhcprelay running?

[aeschma]

No relay is running. DHCP and DHCP6 server is disabled on this Interface.

[Patrick M. Hausen]

Then something must be wrong at layer 2 with your switches - the trunks - the VLAN configuration ...

OPNsense will definitely not forward DHCP requests from one interface to another one if there is no DHCP relay.

[HenrikHenkel]

Well... Thats unfortunate.

In my setup there is MDNS and SSDP traffic leaking from LAN to the WAN site.
No MDNS-repeater setup, no special NAT rules, not even VLANs.

I also tried blocking this traffic with firewall rules, but it seems to only affect about 80% of the packets. Some still get through to the WAN interface and cause a broadcast/multicast loop.

[aeschma]

Ok.

In this case, shouldn't there also be a broadcast flood in the LAN or in one of the LAN VLAN? Configuration for WAN vlan is exactly the same like lan vlan configuration. I will delete this evenig the wan vlan and rebuild it from scratch....

To be sure... I checked under Services -> DHCP Relay whether a dhcp relay exists or not.

[aeschma]

@HenrikHenkel is IGMP Snooping or Multicast DNS traffic enabled on vlan configuration?

[HenrikHenkel]

There are no VLANs configured. All dumb switches.

Each network has its own interface in OPNSense.