Show Posts
1
Hardware and Performance / Bandwidth cut in half when traversing system but direct bandwidth test is fine
« on: April 01, 2024, 12:27:00 am »
Preamble
Hi there :-),
i have a problem with my opnsense setup that is strange to me, maybe someone has in idea on where to poke further.
The Situation/Problem
Given is a opnsense box on decent hardware (see hardware) that is supposed to route traffic from interface1 to interface2 or vice versa at nearly link speed (1G).
Launching iperf from the opnsense machine testing the connection to each system (ubuntu test machine and windows test machine) it reaches "at least high" speed.
Observation
It seems that when traffic is routed through opnsense, my bandwidth is cut in half.
Opnsense System
MTU related
Opnsense uses the following MTUs:
What has been done so far
I did not mess with the checkbox "VLAN Hardware Filtering" yet.
Special notes
Some results
From opnsense to windows test machine
From opnsense to ubuntu test machine
From windows test machine to ubuntu test machine
My questions
Update 2024-04-05
I would be greatfull for any advice. After days been spent on this problem i am losing my mind.
Kind regards and thanks in advance
SimonGuy
Hi there :-),
i have a problem with my opnsense setup that is strange to me, maybe someone has in idea on where to poke further.
The Situation/Problem
Given is a opnsense box on decent hardware (see hardware) that is supposed to route traffic from interface1 to interface2 or vice versa at nearly link speed (1G).
Launching iperf from the opnsense machine testing the connection to each system (ubuntu test machine and windows test machine) it reaches "at least high" speed.
| From | To | Iperf result |
| opnsense | ubuntu | ~833 Mbits/sec |
| opnsense | windows | ~653 Mbits/sec |
| windows | ubuntu | ~302 Mbits/sec |
Observation
It seems that when traffic is routed through opnsense, my bandwidth is cut in half.
Code: [Select]
ix0,ix2
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
┌──────────────────────────────────────────────────────┐
│ opnsense │
│ │
┌──────────────────────┐ │ │
│An Ubuntu test machine│ ├────────────┐ │
│192.168.77.47 │ │ ├────────────────┐ ┌─────────────┐ │
│ │ Tagged ┌──────────┐ Tagged ├───┐ │ lagg0_vlan335 │◄───►│interface1 │ │
│CPU use < 25% │◄────────────►│ │◄───────────►│ix0│ ├────────────────┘ │192.168.77.51│ │
└──────────────────────┘ 1G │ Multiple │ 10G ├───┘ │ └─────────────┘ │
│ Mikrotik │ │ lagg0 │ │
┌──────────────────────┐ Tagged │ Switches │ Tagged ├───┐ │ ┌─────────────┐ │
│A Windows test machine│◄────────────►│ │◄───────────►│ix1│ ├────────────────┐ │interface2 │ │
│10.10.10.76 │ 1G └──────────┘ 10G ├───┘ │ lagg0_vlan1111 │◄───►│10.10.10.1 │ │
│ │ Less then │ ├────────────────┘ └─────────────┘ │
│CPU use < 25% │ 5% CPU use ├────────────┘ │
└──────────────────────┘ │ │
│ All CPUs are idle during transmission │
└──────────────────────────────────────────────────────┘
Opnsense System
| CPU type | Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz (4 cores, 8 threads) |
| Memory usage | 11 % ( 918/8044 MB ) { ARC size 229 MB } |
| Version | OPNsense 24.1.4-amd64 |
| Network Card (onboard) | Intel(R) X552 (SFP+) |
| Mainboard | Supermicro X10SDV-TP8F |
MTU related
Opnsense uses the following MTUs:
| Interface | MTU |
| ix0 | 1470 |
| ix1 | 1470 |
| lagg0 | 1470 |
| lagg0_vlan335 | 1300 |
What has been done so far
- Messed with the MTU (should not be a problem has the opnsense can communicate fine with ubnutu?)
- Checked and unchecked "Disable reply-to"
- Used "pfctl -d" temporarily to disable the firewall
- Unchecked all three of "Hardware CRC", "Hardware TSO" and "Hardware LRO"
I did not mess with the checkbox "VLAN Hardware Filtering" yet.
Special notes
- The system is in a HA mode with an identical other node
- The system has a lot (20+) Interfaces on vlans all on lagg0
- An IPSEC VDI Tunnel is again slower, even when using local ehternet only connection (just a side node, its what got the investigation started...)
- Checked CPU load on all related switches, all are sube 10% and bandwidth seems no issue aswell
- The system has only a few fw rules
- IPERF Options used (-w 64KB), tests added to the post are with "-t 2" but to shorten the output. longer tests show similar results
Some results
From opnsense to windows test machine
Code: [Select]
Connecting to host 10.10.10.76, port 6666
[ 5] local 10.10.10.2 port 6661 connected to 10.10.10.76 port 6666
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.03 sec 105 MBytes 853 Mbits/sec 0 209 KBytes
[ 5] 1.03-2.00 sec 98.6 MBytes 856 Mbits/sec 0 209 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-2.00 sec 204 MBytes 855 Mbits/sec 0 sender
[ 5] 0.00-2.00 sec 204 MBytes 855 Mbits/sec receiver
From opnsense to ubuntu test machine
Code: [Select]
Connecting to host 192.168.77.47, port 6666
[ 5] local 192.168.77.52 port 7437 connected to 192.168.77.47 port 6666
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 109 MBytes 914 Mbits/sec 0 3.00 MBytes
[ 5] 1.00-2.00 sec 108 MBytes 903 Mbits/sec 0 3.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-2.00 sec 217 MBytes 909 Mbits/sec 0 sender
[ 5] 0.00-2.00 sec 217 MBytes 909 Mbits/sec receiver
From windows test machine to ubuntu test machine
Code: [Select]
Connecting to host 192.168.77.47, port 6666
[ 4] local 10.10.10.76 port 57609 connected to 192.168.77.47 port 6666
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 47.6 MBytes 399 Mbits/sec
[ 4] 1.00-2.00 sec 49.0 MBytes 411 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-2.00 sec 96.6 MBytes 405 Mbits/sec sender
[ 4] 0.00-2.00 sec 96.6 MBytes 405 Mbits/sec receiver
My questions
- Why can the system talk to the test machines so fast but traffic that traverses the opnsense is cut in half?
- Can i rule out MTU settings as the opnsense system can communicate fine when doing so directly? (it should use the same ports with the same vlans and so on... so it must be fine - right?)
- Is there any reason to doubt the network card or the driver when again, it can work fine communicating directly? I suppose not(?)
Update 2024-04-05
- Fiddling with the flow control sysctl setting did not help with the problem.
- Changing to different DAC cables CISCO, HP, Huawei (recommenden by our supplier for this card/board)
Code: [Select]
dev.ix.0.fc = 0
dev.ix.1.fc = 0I would be greatfull for any advice. After days been spent on this problem i am losing my mind.
Kind regards and thanks in advance
SimonGuy