Sensei and bufferbloat

[fabianodelg]

Hi all

since I installed Sensei on my APU2 I did notice that I started to experience quite a serious bufferbloat; I did few test and I could see that if I would execute a speedtest while pinging 1.1.1.1 I could see that there is a very high latency (>300ms) and, packet loss (random) -I have a 100/10 wan link-

DSLreport, is giving me an F http://www.dslreports.com/speedtest/69317047

I have FQ_Codel implemented but it seems that there's no too much that can be done to resolve.

Switching Sensei to passive mode, the ping latency disappear (during the speedtest, <30ms) and DSLreport, now give me an A.

Is this a case of too underpowered hardware or there's any tweak that can be done to improve the Sensei performance?

My APU2 run on a AMD GX-412TC SOC (4 cores) with 4 GB RAM; the BIOS has been upgraded to allow 1.4 GHz

Thanks
F

[mb]

Hi @fabian,

APU is quite a weak hardware but I think you should be able to attain higher speeds. What happens if you put Sensei into bypass mode?

[RamSense]

Im am running opnsense / sensei on Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz (4 cores) and 16GB RAM.
I have no traffic shaping pipes,gues or rules.
I can reach full speed (600 mbit/s download and 40 mbit/s upload) on speedtest.net and dslreports. But see also bufferbloat. It does not change with or without sensei.

with sensei on:
Overall: A
Bufferbloat: C
quality: A+

when putting sensei in bypass, same result
when shutting down sensei, also the same:

Overall: A
Bufferbloat: C
quality: A+

I think my hardware is more than enough, so can this bufferbloat improve with FQ_Codel? Is there a guide for this?

[fabianodelg]

Hi @Ramsense

here's the guide; apply that and you should see loads of improvements....

https://maltechx.de/en/2021/03/opnsense-setup-traffic-shaping-and-reduce-bufferbloat/

Best
F

[fabianodelg]

Hi @fabian,

APU is quite a weak hardware but I think you should be able to attain higher speeds. What happens if you put Sensei into bypass mode?

Hi @mb

thanks for your answer. I do get full speed with both speedtest and dslreport (with or without Sensei): the problem is the latency. With Sensei in Routing mode (L3+blocking) during the above test the latency goes sky high and I also experience  packet loss.

With Sensei in Passive mode (which I suppose is what you call bypass), I still have full speed but, this time, with low latency during the speed test (which is what we want...). Sensei is monitoring 1 interface out of 2.

To be notice that I do use the traffic shaper on the WAN interface to improve the bufferbloat.

Thanks for you help,

Best
F

[RamSense]

@fabianodelg thank you for your link and info!
I will try it and see if I can get better results...

[RamSense]

Just followed you guide (thanks again for that url) and the results are:
(With sensei and suricata on)

Overall: A+
Bufferbloat: A
Quality: A+

speed was 550 instead of the 600 earlier, but there is more traffic use now, so I will test it later when I am the only one using the internet.
So your guide works great!

[mb]

Hi @fabian,

Bypass mode is not the same with Passive Deployment Mode. In L3 Mode, bypass allows you to temporarily bypass any Sensei packet processing while you're still running Sensei. In this mode, Sensei basically behaves like a dummy bridge.

Head to Sensei -> Status and locate "Services". There you'll see "Enter Bypass Mode" option where you see the "Sensei Packet Engine" information.

This test is to see if the additional latency is induced by netmap or the engine itself.

[fabianodelg]

Just followed you guide (thanks again for that url) and the results are:
(With sensei and suricata on)

Overall: A+
Bufferbloat: A
Quality: A+

speed was 550 instead of the 600 earlier, but there is more traffic use now, so I will test it later when I am the only one using the internet.
So your guide works great!

Glad to hear! Just consider that the shaper will cap a bit of the top end bandwidth, that's why you would not probably see the full speed during the test.

The most important thing, imho, is to have a snappy internet; this is what the shaper is helping with (in this case).

Best
F

[fabianodelg]

Bypass mode is not the same with Passive Deployment Mode. In L3 Mode, bypass allows you to temporarily bypass any Sensei packet processing while you're still running Sensei. In this mode, Sensei basically behaves like a dummy bridge.

Head to Sensei -> Status and locate "Services". There you'll see "Enter Bypass Mode" option where you see the "Sensei Packet Engine" information.

This test is to see if the additional latency is induced by netmap or the engine itself.

Thanks @mb, I didn't notice that... and yes I will run the test and post the result.

Just a question: in Passive mode (so with Sensei on but not blocking any traffic) I cannot see any packet loss nor any increase of latency (ed in fact, there is no bufferbloat); what passive mode does?

Thanks and best
F

[mb]

Hi @fabian,

Passive mode uses the pcap interface, where "a copy" of the packet is handed over to Sensei while it's traveling to its destination. In this mode, Sensei runs in parallel to the packet flow.

L2/L3 modes uses the netmap interface where the packet is first handed over to Sensei and according to the policy in place Sensei decides whether to switch the packet or not. In this mode, Sensei in running in between the ethernet adapter and the host operating system.

In the latter, generally, we measure additional latency introduced by Sensei around 0.2ms. In the former case, since we're not in the middle of the packet flow, additional latency is zero. Having said that, because of this you cannot do filtering. It's useful for reporting.

[fabianodelg]

Hi @fabian,

Bypass mode is not the same with Passive Deployment Mode. In L3 Mode, bypass allows you to temporarily bypass any Sensei packet processing while you're still running Sensei. In this mode, Sensei basically behaves like a dummy bridge.

Head to Sensei -> Status and locate "Services". There you'll see "Enter Bypass Mode" option where you see the "Sensei Packet Engine" information.

This test is to see if the additional latency is induced by netmap or the engine itself.

Hi @mb

I did the bypass test and no appreciable difference when Sensei is in Passive mode (so no high latency and the bufferbloat is A)

Would that mean that the culprit is around the scan engine? As you say, if the additional latency is only 0.2ms, I think this maybe more a case of 'underpowered' hardware. I run a 100/10 WAN link but I suppose thing will get much worse with higher bandwidths (and Sensei monitoring multiple lan interfaces).

I used to use a NUC (Intel i7 8 core with 32 GB RAM) for OPNsense never had a problem throwing anything at it; the only reason why I reverted to the APU is that the NUC has a single network card and I added the WAN link with a USB to Ethernet connection; despite the chipset was fully supported by freeBSD, everynow and then the the adapter was disconnected from the system with the obvious consequences...

The APU is undoubtedly slower than a NUC but it is stable as a rock.