Traffic being dropped occasionally during inter-VLAN routing

[Knogle]

Hey friends, how are you doing.

I have migrated my network from using OpenWrt to OPNsense and i am extremely satisfied.

Unfortunatly i am encountering some weird behaviour.

I have got a bunch of servers, accessible through my management VLAN 110 in 172.20.32.0/19.

My trusted LAN network is in VLAN 3 in 192.168.3.0/24.

Now the interesting part. A lot of times now, when i establish SSH connections from one of my PCs in 192.168.3.0/24, to one of my servers in 172.20.32.0/19 the connection is dropped after some random amount of time. It occurs very frequently, and after a bunch of seconds.

Some example, of SSH connection going down, while being logged into my TrueNAS machine.

Code: [Select]

ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
root@truenas[~]# debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: channel 0: free: client-session, nchannels 1
Connection to 172.20.32.5 closed by remote host.
Connection to 172.20.32.5 closed.
Transferred: sent 2992, received 47976 bytes, in 94.1 seconds
Bytes per second: sent 31.8, received 510.1
debug1: Exit status -1
Whenever a connection dies to any of my servers its aborted with closed by remote host message.

Even SSH ZFS replications on TrueNAS, to a different subnet, loose their SSH connection the same way.

When SSHing inside of one subnet, so not going through the OPNsense appliance it works flawlessly.


Also another example. When trying to iperf3 to a different subnet, there is simply starvation. No traffic at all. Going through the directly connected subnet it works fine.

Code: [Select]

chairman@fedora:~$ iperf3 -c 192.168.200.5
Connecting to host 192.168.200.5, port 5201
[  5] local 192.168.3.22 port 50422 connected to 192.168.200.5 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   386 KBytes  3.16 Mbits/sec    2   1.41 KBytes       
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   386 KBytes   316 Kbits/sec    5             sender
[  5]   0.00-10.00  sec  65.0 KBytes  53.3 Kbits/sec                  receiver

iperf Done.

Code: [Select]

chairman@fedora:~$ iperf3 -c 192.168.3.5
Connecting to host 192.168.3.5, port 5201
[  5] local 192.168.3.22 port 41500 connected to 192.168.3.5 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   946 Mbits/sec   12    153 KBytes       
[  5]   1.00-2.00   sec   111 MBytes   933 Mbits/sec    9    158 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   939 Mbits/sec    6    212 KBytes       
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec   11   96.2 KBytes       
[  5]   4.00-5.00   sec   111 MBytes   933 Mbits/sec   12    204 KBytes       
[  5]   5.00-6.00   sec   111 MBytes   932 Mbits/sec    9    136 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   940 Mbits/sec   10    161 KBytes       
[  5]   7.00-8.00   sec   111 MBytes   932 Mbits/sec   11    168 KBytes       
[  5]   8.00-9.00   sec   111 MBytes   932 Mbits/sec   13    171 KBytes       
[  5]   9.00-10.00  sec   111 MBytes   933 Mbits/sec    9    184 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   936 Mbits/sec  102             sender
[  5]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec                  receiver

iperf Done.


What is wrong here? Did anyone encounter some similar behaviour before?

Thanks in advance!