Messages

I'm waiting to get this one which is slightly more powerful and is with 4*Intel i226-V 2.5G
I'll also try to put OPNSense on it and see if it works.

I'm on Xfinity's Gigabit Xtra plan which is supposed to be 1200Mbps. With 40Mbps upload speed. I have codel_fq on the egress to keep the latency in check as the outbound is easy to saturate. I don't have traffic shaper running on the ingress. The download speed maxes out at around 1400Mbps. Note the high latency which means that the modem link is saturated. The connection between the firewall and the modem is 2.5Gbps so it still has capacity.

Code Select Expand


speedtest

   Speedtest by Ookla

      Server: Race Communications - San Francisco, CA (id: 8228)
         ISP: Comcast Cable
Idle Latency:    18.01 ms   (jitter: 1.18ms, low: 17.09ms, high: 19.35ms)
    Download:  1423.36 Mbps (data used: 2.2 GB)                                                   
                 33.47 ms   (jitter: 4.75ms, low: 9.96ms, high: 159.69ms)
      Upload:    38.87 Mbps (data used: 37.6 MB)                                                   
                 17.45 ms   (jitter: 2.68ms, low: 11.94ms, high: 24.88ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/1bd53d7b-803b-40f4-ad0d-40bc9ce9838a


[net.link.ether.inet.max_age=120]

I had a continuous problems with monitoring my WAN gateway when using the Xfinity provided modem when the modem is set in bridge mode. The connection would start fine, work for a few min and then the modem will stop responding. Took me a bit of time to figure out what was wrong.  Seems that the Technicolor modem CGM4331COM that Xfinity supplies stops responding if there isn't an ARP request for the modem's IP within preceding 5 min. This is very weird limitation as the ethernet address of the modem doesn't change so the response of each ARP request gives you back the same ethernet address each time. The problem doesn't seem to happen with Linux and Windows as by default these two OSes have ARP cache entry lifetime of less than a minute. For FreeBSD the default is 20 min. Once I found out the cause of the problem, the fix for the OPNSense firewall was to simply add net.link.ether.inet.max_age=120 in the tunables. As I posted previously - the Motorola MB8611 doesn't have this problem nor has any other modem that I have seen before. A very weird and undocumented behavior that should be fixed.
Perhaps for OPNSense we can set by default net.link.ether.inet.max_age to 1 or 2 minutes so people don't have to deal with these issues.

I seem to have similar issue https://forum.opnsense.org/index.php?topic=27433.0
I have two WANs, both doing gateway monitoring. One is having the problem. The other seems fine. When it happens, the interface that is affected shows double ICMP replies of the dpinger pings.

Code Select Expand

tcpdump -n -i cxl1 icmp
...
23:51:43.962528 IP 98.51.182.15 > 4.2.2.3: ICMP echo request, id 33608, seq 0, length 8
23:51:43.981125 IP 4.2.2.3 > 98.51.182.15: ICMP echo reply, id 33608, seq 0, length 8
23:51:43.981169 IP 4.2.2.3 > 98.51.182.15: ICMP echo reply, id 33608, seq 0, length 8


[tcpdump]

I'm having a problem on one of my internet connections (I have two) when I enable gateway monitoring.
When I enable gateway monitoring, I get either 100% loss or periods of few min with a loss and few min with no loss. I figured out that the problem seems to be related to the host being monitored added to the routing table. I see this row added when I enable gateway monitoring (the host being pinged is 4.2.2.3).

Code Select Expand

Routing tables
Internet:
Destination        Gateway            Flags     Netif Expire
...
4.2.2.3            98.51.132.1        UGHS       cxl1
...

When that entry is in the routing table, pinging the host doesn't work

Code Select Expand


ping 4.2.2.3
PING 4.2.2.3 (4.2.2.3): 56 data bytes
^C
--- 4.2.2.3 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss

I also tried tcpdump to see if the ping is working and apparently each reply is duplicated when the route to the host being monitoried is in the routing table

Code Select Expand


tcpdump -n -i cxl1 icmp
...
20:02:51.431991 IP 98.51.135.187 > 4.2.2.3: ICMP echo request, id 18212, seq 963, length 8
20:02:51.460338 IP 4.2.2.3 > 98.51.135.187: ICMP echo reply, id 18212, seq 963, length 8
20:02:51.460349 IP 4.2.2.3 > 98.51.135.187: ICMP echo reply, id 18212, seq 963, length 8

This is what dpinger looks like for this interface

Code Select Expand


/usr/local/bin/dpinger -f -S -r 0 -i WAN_DHCP -B 98.51.135.187 -p /var/run/dpinger_WAN_DHCP.pid -u /var/run/dpinger_WAN_DHCP.sock -C /usr/local/etc/rc.syshook monitor -s 1s -l 2s -t 60s -A 1s -D 500 -L 20 -d 0 4.2.2.3


Given that dpinger is using explicit interface, it isn't clear why that route has to be added. I looked at the code (/usr/local/etc/inc/plugins.inc.d/dpinger.inc)and found this:

Code Select Expand


            /*
             * If the gateway is the same as the monitor we do not add a
             * route as this will break the routing table.
             * Add static routes for each gateway with their monitor IP
             * not strictly necessary but is an added level of protection.
             */
            if (is_ipaddrv4($gateway['gateway']) && $gateway['monitor'] != $gateway['gateway']) {
                log_error("Adding static route for monitor {$gateway['monitor']} via {$gateway['gateway']}");
                system_host_route($gateway['monitor'], $gateway['gateway'], false, true);
            }

The comment seems to indicate as well that adding the route is redundant. I commented the section and tried again.
This seems to have solved the problem. tcpdump shows again single reply for each request

Code Select Expand

tcpdump -n -i cxl1 icmp
...
20:09:37.074603 IP 98.51.135.187 > 4.2.2.3: ICMP echo request, id 8970, seq 12, length 8
20:09:37.111082 IP 4.2.2.3 > 98.51.135.187: ICMP echo reply, id 8970, seq 12, length 8

pinging the host being monitored also works. It does require however to specify source address given that there isn't a route. When the host being monitored is in the routing table, ping with a source address doesn't work as well as ping without it.

Code Select Expand


ping -S 98.51.135.187 4.2.2.3
PING 4.2.2.3 (4.2.2.3) from 98.51.135.187: 56 data bytes
64 bytes from 4.2.2.3: icmp_seq=0 ttl=57 time=18.534 ms
64 bytes from 4.2.2.3: icmp_seq=1 ttl=57 time=17.202 ms


So, I guess the question here is - can we remove that code that adds static route given that dpinger uses explicit interface?
I can't say why that is a problem. I'm just observing that with the static route it doesn't work in my case and without it - it does.