Latency vs typical ISP router or (pro/con)sumer microtik

[planetf1]

I'm thinking of running opnsense on a Intel N100, Intel 226V, 16GB ram.
Most likely virtualized, could also be bare metal.

I rarely see latency discussed anywhere much, but I'm interested to know what other's experiences have been with opnsense when compared to typical ISP routers (fritz!box, eero for example, or any!), or even a microtik.

Clearly the x86 approach has lots more horsepower, but less hw optimization

For example a simple ping to the router from lan, or more interestingly a comparison on first hop etc vs one of the other alternatives?

Similarly how is jitter - the little dedicated boxes (I have a fritz7530) aren't powerful, but I've found them very consistent. (put aside any affect from additional workload for now)

[Seimus]

There is not much to discuss about latency when pinging a directly attached device in your LAN.

The stronger the CPU the faster the response.

OPNsense is pretty consistent, I am measuring my Path to OPN and thru it, as well all Telco HOPs in order to identify when there is a problem with connectivity.

Here you can see OPN is pretty consistent. Additionally each 30mins runs a speed test from a server I have to a remote destination. As you can see OPNsense is running without hiccup even if it does have high traffic loads to transfer. I tested latency on OPN as well on full load of 2Gbit/s LAGG, values are the same.



Regards,
S.

[planetf1]

Many thanks for sharing.

What hardware approx. did you use?

For lan I get ~ 0.31ms through 2 dumb switches to router which seems decent.

I guess my question arose from a surprise higher latency from one particular piece of ISP equipment (~+1-1.2 ms, and a little more jitter say 0.5-0.8ms) over 1000s of icmp tests to known internet destinations.. In that case a simple swap in/out of hardware gives me high confidence it was a real problem. I'm hoping/wanting not to encounter that

But there are so many variables it's hard to really know
 - bare metal vs passthru vs virtualized network interfaces (my idea will be virtualized with was passthrough & lan bridged) - here the virtualization drivers etc are a big factor. Virtualization allows more flexibility esp. with filtering/dns and any other *light* workload, perhaps haproxy. But acceptable fallback will be bare metal

[Seimus]

Many thanks for sharing.

What hardware approx. did you use?

N5105 - 4x i226-V | Patriot 2x8G 3200 DDR4 | EVO 980 500G

I guess my question arose from a surprise higher latency from one particular piece of ISP equipment (~+1-1.2 ms, and a little more jitter say 0.5-0.8ms) over 1000s of icmp tests to known internet destinations.. In that case a simple swap in/out of hardware gives me high confidence it was a real problem. I'm hoping/wanting not to encounter that

~+1-1.2 ms, with jitter 0.5-0.8ms is still fine for LAN environment.

But there are so many variables it's hard to really know
 - bare metal vs passthru vs virtualized network interfaces (my idea will be virtualized with was passthrough & lan bridged) - here the virtualization drivers etc are a big factor. Virtualization allows more flexibility esp. with filtering/dns and any other *light* workload, perhaps haproxy. But acceptable fallback will be bare metal

I am running Baremetal, Reason is I wanted beefy system to handle my current max throughput >2GBit/s and all future feature set and implementations. Also keep in mind if you go the virtualization way, let say proxmox, you are adding additional complexity. So have a good understanding of proxmox & virtualisation.

Regards,
S.

[Raketenmeyer]

I replaced my Fritz! box at home with a DEC740 a few years ago. There was a significant improvement in latency here, but I no longer have any exact data from that time. The busier the network was here at home, the greater the advantage of the DEC740.

Here is a screenshot (quality/WAN) of our DEC3850 from the data center. Monitoring goes to the ISP gateway.

[cookiemonster]

@Seimus. I just had some work-related MS Teams calls being disrupted, so I installed and configured smokeping to rule out the ISP. DNS and Ping to external sites are being graphed, that's fine.
How are you measuring your Path to OPN and thru it if you don't mind?

[Seimus]

@cookiemonster

Sure its not problem to share but let me first introduce what tools I use.

1. Uptime-Kuma
- I started to use this as a replacement for blackbox, due to the fact that blackbox had some performance issues and was starting to piss me off a bit.
https://github.com/louislam/uptime-kuma

2. Prometheus + Grafana
- Uptime-Kuma has very good front end web UI you can see the picture from the post above that's how it looks like. But additional to have more side to side comparison in one graph I scrub the data from uptime-kuma via Prometheus and sent the to grafana.
https://github.com/grafana/grafana
https://github.com/prometheus/prometheus


3. Speedtest-exporter by MiguelNdeCarvalho
- This is  a very nice tool that tests WAN speed, jitter & latency towards the closest (or by you set) OOkla server.
https://github.com/MiguelNdeCarvalho/speedtest-exporter

All the above runs in docker container, under a specific stack I call internet monitoring (+ other containers for monitoring but this 3 above mentioned are main for this situation.)

The outcome of this looks like this:



I monitor the following things in the following order with these types of probes >
A. DNS - selfhosted recursive Pihole - Probe type: DNS
B. OPNsense - GW/FW - Probe type: ICMP
C. ISP 1st HOP - Probe type: ICMP
D. ISP 2nd HOP - Probe type: ICMP
E. ISP 3rd HOP - Probe type: ICMP
F. ISP 4th HOP - Probe type: ICMP
G. ISP 5th HOP - Probe type: ICMP
H.1 Control point: OPNsense main webpage - Probe type: HTTPs
H.2 Control point: ISP main webpage - Probe type: HTTPs
H.3 Control point: City main webpage - Probe type: HTTPs
H.4 Control point: Duckduckgo - Probe type: HTTPs

Using trace I identified Public IPs and devices of my telco and mapped their L3 path from my Device to their last HOP right away either before their hit the destination or handover to other Provider.

Logic:
1. If there is an issue on my LAN
- Probes for DNS or OPNsense will either fail or show performance issues (latency, jitter)
- Probes for ISP Path + Destination will show fail or performance issues (latency, jitter)
- Probes for the Control point will fail or show performance issues (latency, jitter)

2. If there is an issue on  ISP path
- Probes for DNS and OPNsense will be OK
- Probes on the L3 hop that has the issue will fail or show performance issues (latency, jitter) as well probes towards the HOPs behind that point will fail or show performance issues (latency, jitter)
- Probes for the Control point will fail or show performance issues (latency, jitter)

3. This is in regards of the Speedtest-exporter
- Runs every 30minutes to the closest OOkla server
- In case speed test cant deliver the contracted BW purchased from the ISP or experiences jitter, latency while other probes are OK, lets say for more then few runs I report it to them as a fault

4. Not my ISP issue
- Probes for DNS and OPNsense will be OK
- Probes for ISP HOPs/nodes will be OK
- Probes for the Control point will fail or show performance issues (latency, jitter)

I don't think is anything special but it works. I had back in the past problem with O2 of constant performance issues. They didn't believe me but the moment I shoved them the graph and exactly identified their Node/HOP the issue 1st appeared they were able to find it and fix it. Same goes for the Internet throughput.


P.S. Uptime-Kuma can let you configure various types of probes depending of what you need if that specific probing destination supports it
P.S.S Uptime-Kuma can sent notification to various Applications, such as Telegram or Discord using a webhook. Or email etc. I have webhooks set, so if a probe goes down I get a notification.



Regards,
S.

[cookiemonster]

thanks for taking the time Seimus. My big bare metal hosts had to be powered down to conserve energy. Electricity in the UK became ridiculously expensive, and they were hosting my similar monitoring stacks. Moved to small factor modern little machines with low power consumption.
Now just a little machine is running a few daemons, like smokeping. The probes are the ones of interest. I'll have a think. With thanks!

[Seimus]

Your are welcome cookiemonster,

In regards of the stacks and electricity. In CZ I think the situation is better than in UK. Yet still when I started building my home network, I had one requirement in my mind that's power_efficiency/performance.

All the devices I use in my home are what I consider power efficient in consumption aka a single device is not taking more than 20W(my personal power budged per 24h/run device). Saying that, for the monitoring stack I use the same RPi I run the DNS server on. RPi4B is powerful enough to run full DNS, NTP, DHCP + a lot of containers I have divided into stacks.

So if you have power consumption awareness you can run the above mentioned by me without any problems on RPi.


Anyway, if you decide to try anything from above or have any additional questions for the monitoring. Feel free to PM me (cause I have the feeling we bit hijacked this thread from owner, sorry about that mate )

P.S. if there would be interest I can start a standalone topic or create a guide on an efficient self-hosted monitoring system. Or even create a github repo with all my stuff in regards of the monitoring.

Regards,
S.

[planetf1]

Interesting posts - especially approve of graphs showing latency in microseconds ;-)