Issue with Bridged Ports in OPNsense Setup

[Combatsatellite]

Hello everyone,

I've encountered a strange issue with my current OPNsense setup and could really use some help.

Previously, I had the same NIC bridged in the exact same way, and it worked as expected. However, now I'm experiencing a problem when using the bridged ports: I can't access services running on devices connected to other ports of the same bridge.

Here's an example:

    My NAS is at 10.0.0.20, and my PC is at 10.0.0.10.
    When both are connected to the same switch behind the bridge, everything works fine—I can access the NAS from the PC.
    But if I plug the NAS or an access point into another bridged port, things go awry:
        The NAS remains reachable from the WAN, but I can no longer access it from my PC.

I've already followed the steps outlined in this guide: https://docs.opnsense.org/manual/how-tos/lan_bridge.html , including setting up the two tunables mentioned there. Those tunables were already configured correctly before the issue arose.

If anyone has experienced a similar problem or has any advice, I'd greatly appreciate your help!

Thanks in advance!

[Combatsatellite]

After further debugging I still was not able to find out what the issue is..

[Monviech (Cedrik)]

You could do a packet capture on the bridge and member interfaces and look at:

- Do arp requests and arp replies work correctly?
- Do the packets move from interface, to bridge, and out of the other interface?
- Do Firewall rules block your packets?

[Combatsatellite]

You could do a packet capture on the bridge and member interfaces and look at:

- Do arp requests and arp replies work correctly?
- Do the packets move from interface, to bridge, and out of the other interface?
- Do Firewall rules block your packets?

I will check it when i have time in the near future (as i will have to re patch some stuff) and report back.

[Patrick M. Hausen]

Post the complete output of "ifconfig" on your OPNsense, please.

[Combatsatellite]

Post the complete output of "ifconfig" on your OPNsense, please.

Here is the full output of ifconfig:

Code Select Expand

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: NIC_Port_1 (opt3)
        options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
        ether 00:1b:21:41:5c:10
        inet6 fe80::21b:21ff:fe41:5c10%igb0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: NIC_Port_2 (opt1)
        options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
        ether 00:1b:21:41:5c:11
        inet6 fe80::21b:21ff:fe41:5c11%igb1 prefixlen 64 scopeid 0x2
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: NIC_Port_3 (opt2)
        options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
        ether 00:1b:21:41:5c:14
        inet6 fe80::21b:21ff:fe41:5c14%igb2 prefixlen 64 scopeid 0x3
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb3: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: NIC_Port_4 (opt5)
        options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
        ether 00:1b:21:41:5c:15
        inet6 fe80::21b:21ff:fe41:5c15%igb3 prefixlen 64 scopeid 0x4
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: Modem_DSL (opt6)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
        ether 4c:cc:6a:b3:d0:39
        inet 192.168.178.21 netmask 0xffffff00 broadcast 192.168.178.255
        inet6 fe80::4ecc:6aff:feb3:d039%em0 prefixlen 64 scopeid 0x5
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0 metric 0 mtu 1536
        options=0
        groups: enc
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
pfsync0: flags=0 metric 0 mtu 1500
        options=0
        maxupd: 128 defer: off version: 1400
        syncok: 1
        groups: pfsync
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33152
        options=0
        groups: pflog
wg0: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1420
        description: HomeWireGuard (opt7)
        options=80000<LINKSTATE>
        inet 10.10.10.1 netmask 0xffffff00
        groups: wg wireguard
        nd6 options=1<PERFORMNUD>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: LAN (lan)
        options=0
        ether 58:9c:fc:10:ff:a0
        inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
        inet 10.10.1.1 netmask 0xffffff00 broadcast 10.10.1.255
        inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igb3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 20000
        member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 2000000
        member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 2000000
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
        description: WAN (wan)
        options=0
        inet 93.130.155.253 --> 62.52.193.57 netmask 0xffffffff
        inet6 fe80::21b:21ff:fe41:5c10%pppoe0 prefixlen 64 scopeid 0xb
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ue0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: WAN_LTE (opt4)
        options=80000<LINKSTATE>
        ether 00:1e:10:1f:00:00
        inet 192.168.8.10 netmask 0xffffff00 broadcast 192.168.8.255
        inet6 fe80::21e:10ff:fe1f:0%ue0 prefixlen 64 scopeid 0xa
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

[Patrick M. Hausen]

So your bridge has got four member interfaces - igb0-3 and the configuration looks good.

I assume em0 is your WAN interface with pppoe0 running on top of that.

From your output only one of the four bridged ports has got any connection. All others have "no carrier". If that is intentional because you are running everything via your switch - ok. If there is a system connected to one of the "no carrier" ports, check the cabling.

If the tunables are in place and the "LAN" or whatever you name it interface is assigned to the bridge, everything looks good.

If you can answer all implicit question with "yes, of course, by the docs" - the next step in my book would be packet traces to watch what is actually happening on the wire.

HTH,
Patrick

[Combatsatellite]

So your bridge has got four member interfaces - igb0-3 and the configuration looks good.

I assume em0 is your WAN interface with pppoe0 running on top of that.

From your output only one of the four bridged ports has got any connection. All others have "no carrier". If that is intentional because you are running everything via your switch - ok. If there is a system connected to one of the "no carrier" ports, check the cabling.

If the tunables are in place and the "LAN" or whatever you name it interface is assigned to the bridge, everything looks good.

If you can answer all implicit question with "yes, of course, by the docs" - the next step in my book would be packet traces to watch what is actually happening on the wire.

HTH,
Patrick

Yes, of course, and by the docs are my answers to the questions.

I won't be able to re patch things today, i will probably get to it tomorrow or the day after.
I will do a packet capture on the LAN and member Interfaces then and come back with the results.

Thanks so far :)

[Combatsatellite]

So your bridge has got four member interfaces - igb0-3 and the configuration looks good.

I assume em0 is your WAN interface with pppoe0 running on top of that.

From your output only one of the four bridged ports has got any connection. All others have "no carrier". If that is intentional because you are running everything via your switch - ok. If there is a system connected to one of the "no carrier" ports, check the cabling.

If the tunables are in place and the "LAN" or whatever you name it interface is assigned to the bridge, everything looks good.

If you can answer all implicit question with "yes, of course, by the docs" - the next step in my book would be packet traces to watch what is actually happening on the wire.

HTH,
Patrick

Here is what i was able to find when searching for the IP i tried to reach on the other port in the packet capture:

[Combatsatellite]

So your bridge has got four member interfaces - igb0-3 and the configuration looks good.

I assume em0 is your WAN interface with pppoe0 running on top of that.

From your output only one of the four bridged ports has got any connection. All others have "no carrier". If that is intentional because you are running everything via your switch - ok. If there is a system connected to one of the "no carrier" ports, check the cabling.

If the tunables are in place and the "LAN" or whatever you name it interface is assigned to the bridge, everything looks good.

If you can answer all implicit question with "yes, of course, by the docs" - the next step in my book would be packet traces to watch what is actually happening on the wire.

HTH,
Patrick

Here is what i was able to find when searching for the IP i tried to reach on the other port in the packet capture:

Is that indicating anything or is more info needed?

[Combatsatellite]

Would it make sense re-installing OPNsense and applying the current configuration?

[Combatsatellite]

I have found out something else, maybe someone has a clue what exactly is going on now..

If i disable the filtering tunable on the Bridge as well it works so it has something to do with the filtering, i did not block anything so maybe it is IDS/IPS or Crowdsec that is at fault here?

ANY help is appreciated!

[Combatsatellite]

I guess one option is to create a rule to allow anything between the interfaces and LAN but as it should work and did before i would like to resolve the original issue..

[Combatsatellite]

Solution was disabling the following rules (screenshot attached) i set up (found after further debugging today).

[Patrick M. Hausen]

Well, if you set up explicit block rules, traffic gets blocked. Why did you do that in the first place?