Messages

1

Hardware and Performance / Re: Simple L3 traffic flood leads to CPU denial of service

« on: March 25, 2023, 07:49:23 pm »

I could find the main culprit and greatly improve the performance :

1. Syslog was the process taking much CPU : I was not aware of a system parameter that enabled logging for default rules, NAT, etc. I was not aware of that because I did not enable logging in my rules.

2. This performance setting was also a game changer : https://docs.opnsense.org/troubleshooting/performance.html#receive-side-scaling

CPU charge is much better controlled now.

I am still experienced some dropped session, so still have some improvements to do, but it's much, much better !

2

Hardware and Performance / Re: Simple L3 traffic flood leads to CPU denial of service

« on: March 25, 2023, 01:16:10 pm »

I am pretty convinced that it's an hardware issue, but as each appliance cost at least 500$, I would like to be sure to get one that works properly before purchasing anything...

I am afraid to spend another 1000$ to get another OPNSense box with more CPU power and get the same issue...

3

Hardware and Performance / Re: Simple L3 traffic flood leads to CPU denial of service

« on: March 25, 2023, 01:07:22 pm »

I am going to give more thought on your comment, but so far I disagree.

To me SYN cookies are more designed to prevent the SYN queue to be filled up, not necessarily the CPU load.
Without SYN cookies, you can do a simple DOS : it means, you can fill up the SYN queue with a single IP address.
It helps the firewall or the server in not maintaining states, and it forces the attacker to respond, using more resources on his side and preventing him from spoofing IP addresses (because he needs to keep track of the connection).

It would not prevent attacks coming from an attacker using a virtually infinite number of IP addresses, when he can spoof them and does not need to care about any reply. He just sends SYN one after another, so SYN cookies are quite irrelevant.

Precisely, hping does not keep track of the connection, it just sends SYN.

Also, I can confirm that SYN cookies are enabled, because when I target an open port on my appliance with hping but without the random source option, the connection gets immediately terminated with "operation not permitted".
Because hping is unable to deal with the cookie.
Without SYN cookies, it proceeds with sending repeated SYN.

4

Hardware and Performance / Re: Simple L3 traffic flood leads to CPU denial of service

« on: March 25, 2023, 10:22:24 am »

Hi,

Yes, SYN cookie has been enabled and it's not related anyway because it's a DDOS.

The attack consist in just a single SYN from many different IPs, the attacker does not care about the reply, so it will never proceed with the TCP conversation ;
Moreover, the attack works against closed ports.

You can tell if the attack reaches your gateway, if you have a traffic peak. Mine, but it will depend on the attacking server, goes up from 1 Mbps at most to a constant 30 Mbps.

If you observe no traffic peak, it means that the DDOS has been block upstream.

If you observe a correlation between the traffic peak and the CPU usage going up to a constant 100%, it means your appliance is vulnerable.

If your CPU is not, or not much impacted, your appliance is not vulnerable.

I get that DDOS can be hard to block, if not impossible when it is based on a higher bandwidth than the one your get.
But in my case, I have 1 Gbps available and the firewall collapses with 30 Mbps, when usually it handles much more bandwitdth with no issue.

It means there is really something wrong in the hardware or software filtering.

That's why I see only three explanations :

1) A misconfiguration on my side, which puts too much load on the CPU ;

2) Weak hardware, either on the CPU or the network card side, which are unable to sustain a high number of connections ;

3) Weak software, by design, where OPNSense is unable to accelerate low level connection and put all the load on the CPU.

It might be a combination of the three...

That's why, even if you don't have a definitive answer, I am interested by your feedback.

If you have a similar configuration and are unable to reproduce the issue, then I have to keep working on my configuration.

Even if you have a different hardware, it's interesting as I might consider replacing my appliance.

If you have the issue, it would confirm that there is something really wrong.

Thanks !

5

Hardware and Performance / Simple L3 traffic flood leads to CPU denial of service

« on: March 24, 2023, 05:34:50 pm »

Hello,

I installed OPNSense on an APU6 with AMD GX-412TC SOC CPU.

It takes a single cloud instance and hping to DDOS the appliance completely. With no TCP port open, just targeting the IP :

hping3 --rand-source -S --flood -p <any port> <ip address>

As a result, the traffic would climb to 30 Mbps (nothing huge) and the CPU would jump and stay at 100% as long as hping3 keeps sending traffic.

I enabled all sort of hardware optimisation that seemed to be available, as well as SYN cookies.

But, as the DDOS works against closed ports, the issue seems to be with L3, and how the kernel handles massive IP requests from random sources (DDOS).

Apparently, there seem to be absolutely no hardware offloading of any sort, which is surprising. L3 traffic should not be compute intensive and traffic should just be quickly dropped.

Did I miss something in the configuration ? Or the APU just sucks for this usage ?

Thank you for any guidance.