Hello everyone,
I've encountered a strange issue with my current OPNsense setup and could really use some help.
Previously, I had the same NIC bridged in the exact same way, and it worked as expected. However, now I'm experiencing a problem when using the bridged ports: I can't access services running on devices connected to other ports of the same bridge.
Here's an example:
My NAS is at 10.0.0.20, and my PC is at 10.0.0.10.
When both are connected to the same switch behind the bridge, everything works fine—I can access the NAS from the PC.
But if I plug the NAS or an access point into another bridged port, things go awry:
The NAS remains reachable from the WAN, but I can no longer access it from my PC.
I've already followed the steps outlined in this guide: https://docs.opnsense.org/manual/how-tos/lan_bridge.html , including setting up the two tunables mentioned there. Those tunables were already configured correctly before the issue arose.
If anyone has experienced a similar problem or has any advice, I'd greatly appreciate your help!
Thanks in advance!
After further debugging I still was not able to find out what the issue is..
You could do a packet capture on the bridge and member interfaces and look at:
- Do arp requests and arp replies work correctly?
- Do the packets move from interface, to bridge, and out of the other interface?
- Do Firewall rules block your packets?
Quote from: Monviech (Cedrik) on November 25, 2024, 03:48:17 PM
You could do a packet capture on the bridge and member interfaces and look at:
- Do arp requests and arp replies work correctly?
- Do the packets move from interface, to bridge, and out of the other interface?
- Do Firewall rules block your packets?
I will check it when i have time in the near future (as i will have to re patch some stuff) and report back.
Post the complete output of "ifconfig" on your OPNsense, please.
Quote from: Patrick M. Hausen on November 25, 2024, 05:15:38 PM
Post the complete output of "ifconfig" on your OPNsense, please.
Here is the full output of ifconfig:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: NIC_Port_1 (opt3)
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether 00:1b:21:41:5c:10
inet6 fe80::21b:21ff:fe41:5c10%igb0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect
status: no carrier
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: NIC_Port_2 (opt1)
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether 00:1b:21:41:5c:11
inet6 fe80::21b:21ff:fe41:5c11%igb1 prefixlen 64 scopeid 0x2
media: Ethernet autoselect
status: no carrier
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: NIC_Port_3 (opt2)
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether 00:1b:21:41:5c:14
inet6 fe80::21b:21ff:fe41:5c14%igb2 prefixlen 64 scopeid 0x3
media: Ethernet autoselect
status: no carrier
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb3: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: NIC_Port_4 (opt5)
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether 00:1b:21:41:5c:15
inet6 fe80::21b:21ff:fe41:5c15%igb3 prefixlen 64 scopeid 0x4
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: Modem_DSL (opt6)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
ether 4c:cc:6a:b3:d0:39
inet 192.168.178.21 netmask 0xffffff00 broadcast 192.168.178.255
inet6 fe80::4ecc:6aff:feb3:d039%em0 prefixlen 64 scopeid 0x5
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0 metric 0 mtu 1536
options=0
groups: enc
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
pfsync0: flags=0 metric 0 mtu 1500
options=0
maxupd: 128 defer: off version: 1400
syncok: 1
groups: pfsync
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33152
options=0
groups: pflog
wg0: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1420
description: HomeWireGuard (opt7)
options=80000<LINKSTATE>
inet 10.10.10.1 netmask 0xffffff00
groups: wg wireguard
nd6 options=1<PERFORMNUD>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: LAN (lan)
options=0
ether 58:9c:fc:10:ff:a0
inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
inet 10.10.1.1 netmask 0xffffff00 broadcast 10.10.1.255
inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: igb3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 20000
member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 3 priority 128 path cost 2000000
member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 2 priority 128 path cost 2000000
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 2000000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
description: WAN (wan)
options=0
inet 93.130.155.253 --> 62.52.193.57 netmask 0xffffffff
inet6 fe80::21b:21ff:fe41:5c10%pppoe0 prefixlen 64 scopeid 0xb
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ue0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN_LTE (opt4)
options=80000<LINKSTATE>
ether 00:1e:10:1f:00:00
inet 192.168.8.10 netmask 0xffffff00 broadcast 192.168.8.255
inet6 fe80::21e:10ff:fe1f:0%ue0 prefixlen 64 scopeid 0xa
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
So your bridge has got four member interfaces - igb0-3 and the configuration looks good.
I assume em0 is your WAN interface with pppoe0 running on top of that.
From your output only one of the four bridged ports has got any connection. All others have "no carrier". If that is intentional because you are running everything via your switch - ok. If there is a system connected to one of the "no carrier" ports, check the cabling.
If the tunables are in place and the "LAN" or whatever you name it interface is assigned to the bridge, everything looks good.
If you can answer all implicit question with "yes, of course, by the docs" - the next step in my book would be packet traces to watch what is actually happening on the wire.
HTH,
Patrick
Quote from: Patrick M. Hausen on November 25, 2024, 10:39:39 PM
So your bridge has got four member interfaces - igb0-3 and the configuration looks good.
I assume em0 is your WAN interface with pppoe0 running on top of that.
From your output only one of the four bridged ports has got any connection. All others have "no carrier". If that is intentional because you are running everything via your switch - ok. If there is a system connected to one of the "no carrier" ports, check the cabling.
If the tunables are in place and the "LAN" or whatever you name it interface is assigned to the bridge, everything looks good.
If you can answer all implicit question with "yes, of course, by the docs" - the next step in my book would be packet traces to watch what is actually happening on the wire.
HTH,
Patrick
Yes, of course, and by the docs are my answers to the questions.
I won't be able to re patch things today, i will probably get to it tomorrow or the day after.
I will do a packet capture on the LAN and member Interfaces then and come back with the results.
Thanks so far :)
Quote from: Patrick M. Hausen on November 25, 2024, 10:39:39 PM
So your bridge has got four member interfaces - igb0-3 and the configuration looks good.
I assume em0 is your WAN interface with pppoe0 running on top of that.
From your output only one of the four bridged ports has got any connection. All others have "no carrier". If that is intentional because you are running everything via your switch - ok. If there is a system connected to one of the "no carrier" ports, check the cabling.
If the tunables are in place and the "LAN" or whatever you name it interface is assigned to the bridge, everything looks good.
If you can answer all implicit question with "yes, of course, by the docs" - the next step in my book would be packet traces to watch what is actually happening on the wire.
HTH,
Patrick
Here is what i was able to find when searching for the IP i tried to reach on the other port in the packet capture:
Quote from: Combatsatellite on November 26, 2024, 06:02:41 PM
Quote from: Patrick M. Hausen on November 25, 2024, 10:39:39 PM
So your bridge has got four member interfaces - igb0-3 and the configuration looks good.
I assume em0 is your WAN interface with pppoe0 running on top of that.
From your output only one of the four bridged ports has got any connection. All others have "no carrier". If that is intentional because you are running everything via your switch - ok. If there is a system connected to one of the "no carrier" ports, check the cabling.
If the tunables are in place and the "LAN" or whatever you name it interface is assigned to the bridge, everything looks good.
If you can answer all implicit question with "yes, of course, by the docs" - the next step in my book would be packet traces to watch what is actually happening on the wire.
HTH,
Patrick
Here is what i was able to find when searching for the IP i tried to reach on the other port in the packet capture:
Is that indicating anything or is more info needed?
Would it make sense re-installing OPNsense and applying the current configuration?
I have found out something else, maybe someone has a clue what exactly is going on now..
If i disable the filtering tunable on the Bridge as well it works so it has something to do with the filtering, i did not block anything so maybe it is IDS/IPS or Crowdsec that is at fault here?
ANY help is appreciated!
I guess one option is to create a rule to allow anything between the interfaces and LAN but as it should work and did before i would like to resolve the original issue..
Solution was disabling the following rules (screenshot attached) i set up (found after further debugging today).
Well, if you set up explicit block rules, traffic gets blocked. Why did you do that in the first place?
Quote from: Patrick M. Hausen on December 12, 2024, 08:25:13 PMWell, if you set up explicit block rules, traffic gets blocked. Why did you do that in the first place?
They where supposed to block my Reolink cameras to access the web (found the rules on some other post either here or on reddit)