Messages

This could be a case of asymmetric routing or maybe, problems because of mixing tagged and untagged VLANs on the same interface.

Since it is Proxmox, you may have also configured both subnets on that...

Bridge mode of course. I do not like double NAT. It is a Leox LXT-010H-D, BTW.

I have verified the gc time for the routes on that system to be 300s:

Code Select Expand

# cat /proc/sys/net/ipv4/route/gc_timeout
300

This would not be a problem in itself, it is just that with this configuration, an ARP request from the ONT will not work.

Thus, reducing the ARP timeout on OpnSense to 300s or below will do the trick.

Was wäre denn Deine Erwartung? Kein reines Failover, sondern Load-Balancing? Der Failover wählt doch das funktionierende Gateway anhand der Priorisierung aus, oder? Selbst wenn Du Load Balancing konfigurierst, bestimmst Du ja nicht selbst, welches Gateway genommen wird.

P.S.: Multi-WAN funktioniert nur für IPv4....

The "Default allow LAN to any rule" is an automatically created rule for your first LAN. Other than that, it is then handled like any other manual rule, i.e. it can have logging enabled or disabled.

The settings in Firewall:Settings:Advanced are for rules "behind the scenes" which are not part of your own rule set and which - apart form some checkboxes - cannot be manipulated from your own ruleset.

I cannot say why this log entry was created, however, there is a small info button on the right of each log line (at least in the live log), if you click on it, you will see the details. Within those details, you will find a link to the "rid" (aka rule id), which gives you the precise rule that led to this log entry.

Ehe Du an der Firewall herumspielst, verkleinere doch die Framed-MTU auf dem NPS, so dass es erst gar nicht zur Fragmentierung kommt - egal, woran es liegt, dass überhaupt eine auftritt (z.B. VPN o.ä.).

One of the potential problems with those 10GbE transceivers is that they get awfully hot, especially when they are in a passively cooled unit and the neigboring SFP+-slot is occupied as well. Under these conditions, higher error rates will occur. I had the same on an X540 adapter, which are made for rack servers with high airflow. Those are not suited for installation in a desktop case.

[did]

It does not matter, because OpnSense cannot reach the IP.


Wow. It seems I have found the problem. The timeout, as it turned out, was exactly 300 seconds. What happens is:

The ARP table entry on OpnSense is valid for 1200 seconds first. My ONT seems to have a timeout of 300 seconds. After 301s, OpnSense still thinks it knows the MAC of the ONT and sends the packet - which is received by the ONT.

It seems my ONT has:

a. No "passive ARP learning".
b. An ARP timeout of only 300 seconds.

Thus, there is no valid ARP entry for answering the request. Even if the ONT does ARP then (of which I am not sure), it will be sent to the WAN interface, not the ONT one. That seems to be the case for some chipsets, so it is probably dependent on the ONT brand. It could probably be circumvented if an intermediate switch was used, but I am not sure.

After reaching the ARP timeout of OpnSense or when the ARP entry is manually removed, it forces an ARP broadcast, which immediately heals the condition.

What did help is setting net.link.ether.inet.max_age=300 on OpnSense.

So, it seems this only happens with certain ONTs that have a short ARP timeout and no passive ARP learning and with this specific setup where there are ambiguities of the two network interfaces on the same physical layer. At least this did never happen with virtual IPs.

I do have a running ISP connection on WAN and I have a MAC set on WAN that is different from the bridge MAC. My symptoms are:

- After a while of inactivity (300 seconds will be enough), a ping does not succeed any more, despite an ARP entry being present and non-expired.
- "arp -d <IP-OF-ONT>" makes the ping work again immediately.
- A continuous ping never stops working.

I have fiddled around with various settings on the bridge and even disabled the igc hardware mac filtering via promiscuous mode, all to no avail.

Nope. Not when the MAC on the WAN interface is manually set. Yet I fear that when I manually set the same MAC on the ONT interface, it might break my internet connection.

I hope I have found it: net.link.bridge.pfil_member=0 could do the trick.

Guess what? The ping stopped working again. It could well be something off with the MAC, because I have to set the MAC on the WAN interface manually and the bridge MAC differs.

Setting net.link.bridge.inherit_mac=1 does not change the bridge MAC immediately, BTW.

P.S.: Now it starts working again out of thin air, still with a different MAC - strange.

FWIW: I tried this approach instead of my usual (working) virtual IP setup.

However, it did not work first - I could not even ping the ONT from OpnSense itself. Only after I disabled my null route for 192.168.0.0/16 did it work - and then, to my complete disbelief, it still worked when I re-enabled the null route!

There is something fishy going on. I have read several reports of incidents where only a reboot made certain things work after settings were changed with 26.7.x.

@lmoore: Now I understand. Interesting that you can use a bridge on an otherwise configure port... the way you do it you create two interfaces, which does the trick. I stand corrected, well done.

Actually, although this is kind of complicated, it should be added to the thread where ONT/modem access for the non-VLAN DHCP case is discussed, just because the "usual" VIP way is risky in case an initial DHCP takes too long.  I cannot seem to find it just right now.

[not]

@lmoore: That will not cut it. The problem is not that the modem cannot be reached.

The problem is that the normal WAN NAT rule does not work, because the translated NAT IP is the virtual IP and not the real WAN IP.

This is, because when the DHCP client on WAN times out on the first try, the virtual IP gets to take the "primary" address slot on the WAN interface. When DHCP later succeeds, this will not change, such that the virtual IP will still be used for normal unbound NAT (not for the NAT rule that you need to reach the modem).

This problem does not manifest when the WAN is on another interface than the modem, like with PPPoE or with a VLAN or when DHCP is fast enough to succeed before the virtual IP gets initialized.

A fix for this is probably non-trivial, because you would have to delete and re-create the virtual IP when the DHCP finally succeeds and that could break other things.

The easy way out would be to skip modem access completely under these conditions. Another way would be if the modem could be configured such that the GUI can be configured on another VLAN than the internet connection. If the ISP wants a VLAN, you should tag it on OpnSense if you can and not leave that to the modem. There seems to be a community firmware for that XGS-PON:

https://pon.wiki/guides/masquerade-as-the-orange-sa-livebox-7-with-the-was-110/#from-the-web-ui

And you probably used the "fix-vlans" setting from here:

https://pon.wiki/guides/masquerade-as-the-att-inc-bgw320-500-505-with-the-was-110/#configure-ont-settings

I think (aka "IDK") that AT&T needs a VLAN. As I said, when you let OpnSense do the tagging, it could work, because you would use different interfaces for ONT and WAN, then.

You probably need the os-realtek-re plugin. Without a network connection, it is a little hard to install it, though.

Correct, I meant OpnSense "snapshots", not ZFS "snapshots" and that you cannot boot them any more, indeed making them obsolete once you upgrade the pool, even if you upgraded the boot files. While I am at it, I might as well delete my old snapshots right now...