Traffic shaping only up to 115 mbit / s

[cocoban78]

Hi everyone,

Machine: Dell PowerEdge T320 Xeon E3-1220 V2 3.10Ghz (4 cores), 16MB RAM, SSD 256Gb
Network CARD: Intel Pro 1000 ET 2x1Gb 1p8d1 (ibg0/1)
System: OPNSense 21.1.5 / FreeBSD 12.1
non-virtual machine
Bandwith from ISP: 500/500 mbit/s

67 workstations in LAN

cpu usage max: 5%, memmory max: 12%

simple configuration:

- port WAN 89.10.10.226 igb0 + 3 virtual IP
- port LAN 10.0.0.1/27  igb1 + 1 virtual IP

NAT Rules LAN: ipv4 <-> *

SHAPER configuration:

download pipe: 60 mbit/s
upload pipe: 30 mbit/s

2 RULES per IP, ex:

Interface:WAN  Protocol:IP  Source:ANY  Destination:10.0.3.33  Target:60
Interface:WAN  Protocol:IP  Source:10.0.3.33  Destination: any Target: 30

and the same for the rest of the ip addresses.

When I connect only one workstation to the switch (Netgear 24x1000 gbit/s) everything works fine.
Movement is properly shaped. When I measure with speedtest.net I always have a result of 60/30 mbit/s.
When I change the limit (pipe) to 400/400 mbit/s, I will get that bandwidth.

The problem begins when other workstations are connected to the network. If they use the Internet and the sum of their bandwidth consumed reaches 70 mbit/s, I cannot get more than 30 mbit/s on speedtest. As the sum of all bandwidths drops to 60mbit/s then my measurement will be 40mbit/s.

My impression is that all machines can reach a maximum of 115mbit/s (measured with iftop ibg0).

Network cards definitely work at 1000mbit/s.

Does anyone have an idea what could be the reason?

Regards,
Marek