slow dhcp on wan -> broken NAT

[Patrick M. Hausen]

Try net.link.bridge.inherit_mac=1

[meyergru]

Guess what? The ping stopped working again. It could well be something off with the MAC, because I have to set the MAC on the WAN interface manually and the bridge MAC differs.

Setting net.link.bridge.inherit_mac=1 does not change the bridge MAC immediately, BTW.

P.S.: Now it starts working again out of thin air, still with a different MAC - strange.

[Patrick M. Hausen]

net.link.bridge.inherit_mac=1 makes the bridge inherit the MAC address of the first member interface the moment it is added.

[meyergru]

Nope. Not when the MAC on the WAN interface is manually set. Yet I fear that when I manually set the same MAC on the ONT interface, it might break my internet connection.

I hope I have found it: net.link.bridge.pfil_member=0 could do the trick.

[lmoore]

I have not set any System Tunables on the test machine..

Using the configurations I provided earlier and only changing the IPv4 address on the ONT interface, I have connected an old DSL modem to the WAN port.

Next, I connected my laptop to the LAN and manually configured the network settings for the 10.200.128.0/24 network.

Confirmed I can ping the DSL modem from OPNsense.

Using my laptop I can ping the DSL modem as well as accessing its web management interface.

This basic DSL modem is configured in Modem (bridge) mode, thus static routes cannot be configured on it so the NAT rule is required.

Black-hole routes are configured in OPNsense.

I don't have a spare DSL service so I can't bring up an address on re0 via DHCP.

Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            10.200.128.1       UGS             em0
10.0.0.0/8         link#3             USB             lo0
10.200.128.0/24    link#2             U               em0
10.200.128.2       link#3             UHS             lo0
127.0.0.1          link#3             UH              lo0
172.16.0.0/12      link#3             USB             lo0
192.168.0.0/16     link#3             USB             lo0
192.168.5.0/24     link#7             U           bridge0
192.168.5.12       link#3             UHS             lo0

ONT interface:

bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: ONT (opt1)
        options=10<VLAN_HWTAGGING>
        ether 58:9c:fc:10:94:50
        inet 192.168.5.12 netmask 0xffffff00 broadcast 192.168.5.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        bridge flags=0<>
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                port 1 priority 128 path cost 55 vlan protocol 802.1q
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>

WAN interface:

re0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: WAN (wan)
        options=2038<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,WOL_MAGIC>
        ether f4:f2:6d:05:4d:60
        inet6 fe80::f6f2:6dff:fe05:4d60%re0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Packet capture on bridge0:

18:57:48.023370 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:57:59.027252 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:06.132540 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:19.492006 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:39.529253 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:40.574491 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:41.636462 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:41.646456 IP6 fe80::f6f2:6dff:fe05:4d60.546 > ff02::1:2.547: dhcp6 solicit
18:58:42.335115 IP 192.168.5.12 > 192.168.5.1: ICMP echo request, id 43521, seq 26, length 40
18:58:42.335503 IP 192.168.5.1 > 192.168.5.12: ICMP echo reply, id 43521, seq 26, length 40
18:58:43.353259 IP 192.168.5.12 > 192.168.5.1: ICMP echo request, id 43521, seq 27, length 40
18:58:43.353541 IP 192.168.5.1 > 192.168.5.12: ICMP echo reply, id 43521, seq 27, length 40
18:58:43.658233 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:44.360620 IP 192.168.5.12 > 192.168.5.1: ICMP echo request, id 43521, seq 28, length 40
18:58:44.360906 IP 192.168.5.1 > 192.168.5.12: ICMP echo reply, id 43521, seq 28, length 40
18:58:45.375716 IP 192.168.5.12 > 192.168.5.1: ICMP echo request, id 43521, seq 29, length 40
18:58:45.376006 IP 192.168.5.1 > 192.168.5.12: ICMP echo reply, id 43521, seq 29, length 40
18:58:46.720254 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:47.345014 ARP, Request who-has 192.168.5.12 tell 192.168.5.1, length 46
18:58:47.345024 ARP, Reply 192.168.5.12 is-at 58:9c:fc:10:94:50, length 28

[meyergru]

I do have a running ISP connection on WAN and I have a MAC set on WAN that is different from the bridge MAC. My symptoms are:

- After a while of inactivity (300 seconds will be enough), a ping does not succeed any more, despite an ARP entry being present and non-expired.
- "arp -d <IP-OF-ONT>" makes the ping work again immediately.
- A continuous ping never stops working.

I have fiddled around with various settings on the bridge and even disabled the igc hardware mac filtering via promiscuous mode, all to no avail.

[lmoore]

Are you pinging your ONT from OPNsense or from a network behind it?

Do you have the NAT rule configured in OPNsense for the ONT network?

[meyergru]

It does not matter, because OpnSense cannot reach the IP.


Wow. It seems I have found the problem. The timeout, as it turned out, was exactly 300 seconds. What happens is:

The ARP table entry on OpnSense is valid for 1200 seconds first. My ONT seems to have a timeout of 300 seconds. After 301s, OpnSense still thinks it knows the MAC of the ONT and sends the packet - which is received by the ONT.

It seems my ONT has:

a. No "passive ARP learning".
b. An ARP timeout of only 300 seconds.

Thus, there is no valid ARP entry for answering the request. Even if the ONT does ARP then (of which I am not sure), it will be sent to the WAN interface, not the ONT one. That seems to be the case for some chipsets, so it is probably dependent on the ONT brand. It could probably be circumvented if an intermediate switch was used, but I am not sure.

After reaching the ARP timeout of OpnSense or when the ARP entry is manually removed, it forces an ARP broadcast, which immediately heals the condition.

What did help is setting net.link.ether.inet.max_age=300 on OpnSense.

So, it seems this only happens with certain ONTs that have a short ARP timeout and no passive ARP learning and with this specific setup where there are ambiguities of the two network interfaces on the same physical layer. At least this did never happen with virtual IPs.