[SOLVED] Inter VLAN routing Iperf results 0.00 bits/s when target has multi LAN

[Underpay6703]

Disclaimer and purpose
I have replicated the same behavior on the other *sense.
Firewall settings were kept on default and an Allow all rule was put in place on all interfaces (to rule out incorrect firewall rules).

What I am after is an explanation and whether a setup like this is non-standard.
I honestly feel pretty silly for not being able to find the explanation, the closest answer I found was on the OPNsense forum https://forum.opnsense.org/index.php?topic=35157.0 but the eventual conclusion was that the person just gave up on VLANs.

Setup
I ask you to follow me on this simple topology found below.
The server has public facing services on VLAN10.
Management of the server and the services is done on VLAN100


The problem: Zero upload speed, but normal download speed

I noticed secure copy (scp) could not upload files to VLAN10 despite being able to connect and use ssh.
Iperf3 then gave me a very weird result: 0.00 bits/s

Code Select Expand

iperf3 -c 192.168.10.10
Connecting to host 192.168.10.10, port 5201
[  5] local 192.168.1.10 port 58322 connected to 192.168.10.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   128 KBytes  1.05 Mbits/sec    2   1.41 KBytes       
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   128 KBytes   105 Kbits/sec    5             sender
[  5]   0.00-10.04  sec  65.0 KBytes  53.1 Kbits/sec                  receiver

iperf Done.

There are three ways I get normal speeds
1: Removing the VLAN100 interface from the dual LAN
This suggests to me that the upload is being (partially) incorrectly routed.
I also noticed that if I changed VLAN100 to a different VLAN (20) on the single-LAN or mult-LAN side (but not both), the speed also returns to normal. Meaning that as long as VLAN100 is present on both devices, the single VLAN device cannot upload to the server's VLAN10.

So multi LAN is not the real issue, the real issue is multi LAN where I try to connect from a device whose VLAN is also setup on the target machine but is not the target for the file transfer.   
What I just don't understand is why.

2: Reverse the iperf3 test (iperf3 -s)
Seems logical given the previous point. The target is the device with only one (V)LAN.

3: Turning off the firewall
Of course very silly, but it at least tells me that there is some traffic rejection going on, though I don't see anything in my Firewall Live View.

Conclusion
As and addition to the disclaimer, I think I just fundamentally misunderstand something that makes this setup act this way. Is it uncommon for a managment VLAN to still have access to a service VLAN?

[Underpay6703]

Solved on netgate forum since I asked the same question there,
https://forum.netgate.com/topic/189287/inter-vlan-routing-iperf-results-0-00-bits-per-second-when-target-has-multi-lan/2
TLDR: asymmetric routing and misconceptions on my end on how management should work.