Messages

[A note on Virtualised setups.]

For anyone coming back to this page looking for a solution, I ran into the same issue described earlier in this thread — broadcast storms and unstable WAN connectivity when both OPNsense firewalls were connected in HA mode with CARP enabled (only on LAN).

A tcpdump on the WAN interfaces showed leakage from the LAN and other VLANs — anything from Dropbox's discovery protocol (on the LAN side) to random broadcasts from IoT devices on separate VLANs LAN Side could unexpectedly trigger a broadcast storm on the WAN side.

After lots of frustration, trial and error, here's what worked  for me:


I created two dedicated, isolated VLANs on my UniFi US-24 switch(but this should work on any L2-managed switch) — one for each WAN connection.

WAN 1:
    •    VLAN 50: "WAN1-Isolated"
    •    Port 1: Modem 1 (access port on VLAN 50)
    •    Port 3: OPNsense A WAN interface (access port on VLAN 50)
    •    Port 5: OPNsense B WAN interface (access port on VLAN 50)

WAN 2:
    •    VLAN 60: "WAN2-Isolated"
    •    Port 2: Modem 2 (access port on VLAN 60)
    •    Port 4: OPNsense A WAN interface (access port on VLAN 60)
    •    Port 6: OPNsense B WAN interface (access port on VLAN 60)

These ports are not trunked, not included in any other VLAN, and have no uplink or routing beyond themselves.



On the switch, I enabled the following:
    •    Storm Control: Limited Unicast, Multicast, and Broadcast to 100 packets/sec
    •    Loop Protection: Enabled on ports 1-6 in my case

This setup effectively creates a "virtual dumb switch" using VLANs — but with better control and built-in protection.

Broadcasts and CARP advertisements are confined within each isolated VLAN, and controlled by the switch.

Again in my case, using a physical "dumb" switch between the modem and firewalls did not solve the issue — likely for the same reason it occurred in the first place  when both firewalls were connected directly to the  modems themselves (through the inbuilt "dumb"  switch).

Since switching to this VLAN-based approach, the system has been stable — no broadcast storms in the last 4 days. Fingers crossed it stays that way!



A note on Virtualised setups.

If running on proxmox, it is imperative to set the machine type to Q35, and set the VIOMMU to INTEL (not default (NONE)). I found that as soon as I changed this setting, the problem returned even with the setup above. I have yet to test if I can do away with the switch completely now, and connect directly to the dumb switches on the modem. Also the WAN cards on opnsense are connected in PCI (direct mode) and removed completely from the host network options.

That should be unique yeah. Or else just use the ${index} which just increments on every iteration.

Thanks again for all the replies and time on this.

Screenshot of the working widget

To put this to bed once and for all, in my specific case the problem was that I have two tunnels with the exact same public key so when this gets executed

Code Select Expand

super.updateTable('wgTunnelTable', [[header, row]], tunnel.publicKey);

The second instance, overwrites the first instance coz they have the same index (tunnel.publicKey).

My workaround I added an index to the forEach loop and  the following :

Code Select Expand


let modifiedPublicKey = `${tunnel.publicKey}-${index}`;

    // Update the HTML table with the sorted rows
    super.updateTable('wgTunnelTable', [[header, row]], modifiedPublicKey);[/font]


So the full function would look like :

Code Select Expand

tunnels.forEach((tunnel, index) => { // 'index' gives you the current position in the loop
    let header = `
        <div style="display: flex; justify-content: space-between; align-items: center;">
            <div style="display: flex; align-items: center;">
                <i class="fa ${tunnel.statusIcon} wireguard-interface" style="cursor: pointer;"
                    data-toggle="tooltip" title="${tunnel.connected ? this.translations.online : this.translations.offline}">
                </i>
                &nbsp;
                <span><b>${tunnel.ifname}</b></span>
            </div>
        </div>`;
    let row = `
        <div>
            <span><a href="/ui/wireguard/general#peers">${tunnel.name}</a> | ${tunnel.allowed_ips}</span>
        </div>
        <div>
            ${tunnel.latest_handshake_fmt
                ? `<span>${tunnel.latest_handshake_fmt}</span>
                   <div style="padding-bottom: 10px;">
                       <i class="fa fa-arrow-down" style="font-size: 13px;"></i>
                       ${tunnel.rx}
                       |
                       <i class="fa fa-arrow-up" style="font-size: 13px;"></i>
                       ${tunnel.tx}
                   </div>`
                : `<span>${this.translations.disconnected}</span>`}
        </div>`;

    // Add '-0', '-1', etc. to the public key using the index
    let modifiedPublicKey = `${tunnel.publicKey}-${index}`;

    // Update the HTML table with the sorted rows
    super.updateTable('wgTunnelTable', [[header, row]], modifiedPublicKey);
});


To further confirm, I have also restored my original configuration with the first interface called WG0, and the widget still works.


Will add this info to the ticket too

Thank you all for your help

J

For the sake of completeness, I have changed WG0 to WG1, WG1 to WG2, and WG2 to WG3, by manipulating the config.xml as suggested above and rebooting. I also had to update the interface assignments from the UI after.

The name change was successful, all connections are up and routing traffic however the widget still shows 2 entries only.

I have deleted the widget and re-created it to no avail

Github ticket @doktonotor is nearly complete, with all the info. I do understand the concept btw. It was just that I did not want to open a ticket just in case, I had some remote case with a wrongly named interface for whatever reason which would not merit an investigation at all.

I will be linking the screenshots in this post in the ticket too.

Thank you all for your replies.

(https://github.com/opnsense/core/issues/7812)

Hi Franco,

Was just gonna post that the header / row seem to be Generated correctly (console.log (header)) just before super.updateTable('wgTunnelTable', [[header, row]], tunnel.publicKey);

returns this :

                <div style="display: flex; justify-content: space-between; align-items: center;">
                    <div style="display: flex; align-items: center;">
                        <i class="fa fa-exchange text-success wireguard-interface" style="cursor: pointer;"
                            data-toggle="tooltip" title="Online">
                        </i>
                        &nbsp;
                        <span><b>wg0 (NordVPN_Instance_XX_1) </b></span>
                    </div>
                </div>
                <div style="display: flex; justify-content: space-between; align-items: center;">
                    <div style="display: flex; align-items: center;">
                        <i class="fa fa-exchange text-success wireguard-interface" style="cursor: pointer;"
                            data-toggle="tooltip" title="Online">
                        </i>
                        &nbsp;
                        <span><b>wg1 (NordVPN_Instance_XX_2) </b></span>
                    </div>
                </div>

                <div style="display: flex; justify-content: space-between; align-items: center;">
                    <div style="display: flex; align-items: center;">
                        <i class="fa fa-exchange text-success wireguard-interface" style="cursor: pointer;"
                            data-toggle="tooltip" title="Online">
                        </i>
                        &nbsp;
                        <span><b>wg2 (NordVPN_Instance_XX_3) </b></span>
                    </div>
                </div>
    }

So it seems to be related to the framework.

If WG0 is not permitted, should I just  change my config, or do you still want to open a ticket to fix ?

Hi,

Just upgraded to OPNsense 24.7.2-amd64. Love the new controls on the widgets.

It seems though that the WireGuard widget is not showing WG0 in the status, see screenshot below.

It states correctly there are 3 tunnels (which are WG0, WG1, WG2) but is only showing status for WG1 and WG2

Hi Check if you have

Firewall -> NAT -> Outbound.

Interface  = The WG Interface

Translation / Target = Interface Address

(I had a similar issue here -> https://forum.opnsense.org/index.php?topic=42147.0)

Answering my own question. I was missing a NAT rule allowing traffic to the WG gateway. Firewall -> NAT -> Outbound.

Interface  = The WG Interface

Translation / Target = Interface Address

Just adding more screenshots of the config for context

I have been banging my head with this one for a few weeks. I am setting up 3 Wireguard connections to nordvpn. Two of which will be used as failover Gatway for Vlan 200, this one works, and the 3rd connection will be used as a sole  gateway for vlan 100.

I have setup the connections, and everything appears to be fine. The failover connection works, but the third connection refuses to route traffic, even though the gateway appears to be up. My hunch is that it has something to do with the tunnel address / gateway configuration.

I have setup 3 peers, as per screen shot using the information I obtained from my keychain access as per this https://www.reddit.com/r/WireGuard/comments/xqz102/extract_nordvpn_wireguard_config_with_macos_no/

I have setup 1 instance per peer, and setup the tunnel address and gateway as per screenshots

I have setup an interface and a gateway for each instance, all gateways appear to be up, and the MT gateways both work and speed is very very good (I am getting > 800Mb/s from a 1Gb/s connection, with minimal cpu usage unlike openvpn).

The US gateway simply refused route any traffic out (from either Vlan 100 or Vlann 200) even though it appears online. The RTT also appears to make sense. An openvpn connection used as gateway works fine.

Any ideas would be greatly appreciated.

Absolutely smooth here too. No issues with LAGG, Vlans, wireguard and openvpn. All working without a hitch. A big WELL DONE ! Not a single issue

Apologies... IT IS WORKING, and did not require a reboot on my end, just needed to bring the interfaces down and back up

As you said LAGG0 still has blank MAC

lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: LAN (lan)
   options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
   ether a8:b8:e0:02:ea:cf
   hwaddr 00:00:00:00:00:00
   inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255
   laggproto lacp lagghash l2,l3,l4
   laggport: igc1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
   laggport: igc2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
   groups: lagg
   media: Ethernet autoselect
   status: active
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


but igc1 and igc2 have proper mac addresses now even in the UI

I confirm it is  working !! Thank you for the super prompt reply

Did some more digging this morning, so it seems that whenever two interfaces are selected for LAGG, their mac address becomes 00:00...:00 ?

lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: LAN (lan)
   options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
   ether 00:00:00:00:00:00
   inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255
   laggproto lacp lagghash l2,l3,l4
   laggport: igc1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
   laggport: igc2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
   groups: lagg
   media: Ethernet autoselect
   status: active
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


-----

igc1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
   ether 00:00:00:00:00:00
   hwaddr a8:b8:e0:02:ea:cf
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc2: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
   ether 00:00:00:00:00:00
   hwaddr a8:b8:e0:02:ea:d0
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active

Not sure if this is normal, or a bug.