LAN bridge isn't passing traffic between physical ports

[agh1701]

Hi All,

I have followed the steps Here LAN Bridge and DHCP works but I cannot access a device on one bridge port from another port.

igc0 is my entire network
igc5 is my WAN

If I plug a PC int0 igc1 it gets DHCP and has access to the internet. I cannot ping a PC/device on igc0.  pinging the PC on igc1 from a PC on igc0 yields the same results.  No ping.

It's like these settings have no effect:

Code Select Expand

net.link.bridge.pfil_member = 0
net.link.bridge.pfil_bridge = 1

Code Select Expand

root@rtr:~ # ifconfig
igc0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: OPT1 (opt1)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
        ether 34:1a:4c:03:bc:79
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: OPT2 (opt2)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
        ether 34:1a:4c:03:bc:7a
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: OPT3 (opt3)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
        ether 34:1a:4c:03:bc:7b
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: OPT4 (opt4)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
        ether 34:1a:4c:03:bc:7c
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: OPT5 (opt5)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
        ether 34:1a:4c:03:bc:7d
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc5: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: WAN (wan)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
        ether 34:1a:4c:03:bc:7e
        inet 69.76.39.223 netmask 0xfffffc00 broadcast 255.255.255.255
        inet6 fe80::361a:4cff:fe03:bc7e%igc5 prefixlen 64 scopeid 0x6
        inet6 2605:a000:dfc0:1d:903a:4278:8616:d7b6 prefixlen 128 pltime 521872 vltime 521872
        media: Ethernet autoselect (2500Base-T <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=1000041<UP,RUNNING,LOWER_UP> metric 0 mtu 1536
        options=0
        groups: enc
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pfsync0: flags=0 metric 0 mtu 1500
        options=0
        maxupd: 128 defer: off version: 1400
        syncok: 1
        groups: pfsync
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33152
        options=0
        groups: pflog
wg1: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1390
        description: TorGuardVPNw1 (opt6)
        options=80000<LINKSTATE>
        inet 10.13.128.121 netmask 0xffffff00
        groups: wg wireguard
        nd6 options=9<PERFORMNUD,IFDISABLED>
wg2: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1390
        description: TorGuardVPNw2 (opt7)
        options=80000<LINKSTATE>
        inet 10.13.110.213 netmask 0xffffff00
        groups: wg wireguard
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: LAN (lan)
        options=100000<NETMAP>
        ether 58:9c:fc:10:ff:80
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::5a9c:fcff:fe10:ff80%bridge0 prefixlen 64 scopeid 0xd
        inet6 2603:6011:e300:8adb:5a9c:fcff:fe10:ff80 prefixlen 64
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igc4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
        member: igc3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        member: igc2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 2000000
        member: igc1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 2000000
        member: igc0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 55
        groups: bridge
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wg0: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1420
        options=80000<LINKSTATE>
        inet 192.168.1.224 netmask 0xfffffff8
        groups: wg wireguard
        nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>

Thanks for any help.

[meyergru]

Look at this:

igc1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: OPT2 (opt2)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
        ether 34:1a:4c:03:bc:7a
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

I wonder how your PC got an IP via DHCP on igc1 when it is not even connected....

[agh1701]

It was connected.

[meyergru]

Can you ping both PCs from OpnSense? Oh, and BTW: Do these machines react to pings at all? Windows sometimes does not.

[meyergru]

And then the most problematic part: Your wg0 interface has the same subnet as bridge0... bad luck.

[agh1701]

Yes.  I can ping both from opnsense.  I can also ping both from my wireguard vpn.  I can ping reliably across my network using switches.  just cant ping or access  across bridge ports

[agh1701]

it is my understanding the bridge driver is a layer 2 bridge so this should work.

[agh1701]

also, I moved my wg0 and ipsec networks to 192.168.10.?

[meyergru]

Sounds strange, it should work. Did you reboot after applying these configurations? Sometimes, some old network settings that had been on the member interfaces, stick. Also, IDK if the tuneables are applied immediately or only on boot.

[agh1701]

Yes, rebooted

updated ifconfig

Code Select Expand

root@rtr:~ # ifconfig
igc0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: OPT1 (opt1)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
        ether 34:1a:4c:03:be:65
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: OPT2 (opt2)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
        ether 34:1a:4c:03:be:66
        media: Ethernet autoselect (2500Base-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: OPT3 (opt3)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
        ether 34:1a:4c:03:be:67
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: OPT4 (opt4)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
        ether 34:1a:4c:03:be:68
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: OPT5 (opt5)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
        ether 34:1a:4c:03:be:69
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc5: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: WAN (wan)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
        ether 34:1a:4c:03:be:6a
        inet 69.133.124.5 netmask 0xfffffe00 broadcast 255.255.255.255
        inet6 fe80::361a:4cff:fe03:be6a%igc5 prefixlen 64 scopeid 0x6
        inet6 2605:a000:dfc0:1d:903a:4278:8616:d7b6 prefixlen 128 pltime 306692 vltime 306692
        media: Ethernet autoselect (2500Base-T <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=1000041<UP,RUNNING,LOWER_UP> metric 0 mtu 1536
        options=0
        groups: enc
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pfsync0: flags=0 metric 0 mtu 1500
        options=0
        maxupd: 128 defer: off version: 1400
        syncok: 1
        groups: pfsync
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33152
        options=0
        groups: pflog
wg1: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1390
        description: TorGuardVPNw1 (opt6)
        options=80000<LINKSTATE>
        inet 10.13.128.121 netmask 0xffffff00
        groups: wg wireguard
        nd6 options=9<PERFORMNUD,IFDISABLED>
wg2: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1390
        description: TorGuardVPNw2 (opt7)
        options=80000<LINKSTATE>
        inet 10.13.110.213 netmask 0xffffff00
        groups: wg wireguard
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: LAN (lan)
        options=0
        ether 58:9c:fc:10:ff:9a
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::5a9c:fcff:fe10:ff9a%bridge0 prefixlen 64 scopeid 0xd
        inet6 2603:6011:e300:8adb:5a9c:fcff:fe10:ff9a prefixlen 64
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igc4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
        member: igc3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        member: igc2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 2000000
        member: igc1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 55
        member: igc0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 55
        groups: bridge
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wg0: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1420
        options=80000<LINKSTATE>
        inet 192.168.2.224 netmask 0xfffffff8
        groups: wg wireguard
        nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
router table

Code Select Expand

root@rtr:~ # netstat -r
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            syn-069-133-124-00 UGS            igc5
dns9.quad9.net     syn-069-133-124-00 UGHS           igc5
10.0.0.1           link#11            UHS             wg1
10.0.0.2           link#12            UHS             wg2
10.13.110.0/24     link#12            U               wg2
10.13.110.213      link#7             UHS             lo0
10.13.128.0/24     link#11            U               wg1
10.13.128.121      link#7             UHS             lo0
69.133.124.0/23    link#6             U              igc5
syn-069-133-124-00 link#7             UHS             lo0
localhost          link#7             UH              lo0
142.93.66.45       10.0.0.2           UGHS            wg2
dns.quad9.net      syn-069-133-124-00 UGHS           igc5
159.65.37.178      10.0.0.1           UGHS            wg1
192.168.1.0/24     link#13            U           bridge0
rtr                link#7             UHS             lo0
192.168.2.224      link#7             UHS             lo0
192.168.2.224/29   link#14            U               wg0
192.168.2.225      link#14            UHS             wg0
192.168.2.226      link#14            UHS             wg0
192.168.2.227      link#14            UHS             wg0
192.168.2.228      link#14            UHS             wg0
192.168.2.229      link#14            UHS             wg0
192.168.2.230      link#14            UHS             wg0
192.168.100.1      syn-069-133-124-00 UGHS           igc5

Internet6:
Destination        Gateway            Flags         Netif Expire
default            fe80::201:5cff:fea UG             igc5
localhost          link#7             UHS             lo0
syn-2603-6011-e300 link#13            U           bridge0
rtr                link#7             UHS             lo0
2605-a000-dfc0-001 link#7             UHS             lo0
fe80::%igc5/64     link#6             U              igc5
fe80::361a:4cff:fe link#7             UHS             lo0
fe80::%lo0/64      link#7             U               lo0
fe80::1%lo0        link#7             UHS             lo0
fe80::%bridge0/64  link#13            U           bridge0
fe80::5a9c:fcff:fe link#7             UHS             lo0

[meyergru]

That all looks O.K. - I would try to set logging on for the default block rules and find if the firewall blocks anything.

Usually, i.e. with a switch, you do not have to have firewall rules to enable traffic on the same network, because the clients can "see" one another and traffic does not even pass OpnSense. But with a bridged setup, the traffic passes over two interfaces, so maybe OpnSense is acting like a transparent bridge then and you need to enable that traffic by some explicit "allow" rules.

I never use these kinds of setups, so I am unsure about this.

[pfry]

[...]
Usually, i.e. with a switch, you do not have to have firewall rules to enable traffic on the same network, because the clients can "see" one another and traffic does not even pass OpnSense. But with a bridged setup, the traffic passes over two interfaces, so maybe OpnSense is acting like a transparent bridge then and you need to enable that traffic by some explicit "allow" rules.[...]

Yup. The usual: all rules on the bridge interface, not on the members; all "layer 3+" traffic over the bridge is filtered by said rules.

[agh1701]

It's intrusion detection.  turning it off make the bridge work.  it's not logging ant rule violations so it might be some sort of netmap bridge driver problem?

[meyergru]

Next Time, please follow advice, in this case https://forum.opnsense.org/index.php?topic=48205.0 point 13.

It is going to Save Time - yours and mine.

[Patrick M. Hausen]

@meyergru link to wrong thread, probably.