Hey friends, how are you doing.
I have migrated my network from using OpenWrt to OPNsense and i am extremely satisfied.
Unfortunatly i am encountering some weird behaviour.
I have got a bunch of servers, accessible through my management VLAN 110 in 172.20.32.0/19.
My trusted LAN network is in VLAN 3 in 192.168.3.0/24.
Now the interesting part. A lot of times now, when i establish SSH connections from one of my PCs in 192.168.3.0/24, to one of my servers in 172.20.32.0/19 the connection is dropped after some random amount of time. It occurs very frequently, and after a bunch of seconds.
Some example, of SSH connection going down, while being logged into my TrueNAS machine.
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
ipmi0: Watchdog set returned 0xd5
root@truenas[~]# debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: channel 0: free: client-session, nchannels 1
Connection to 172.20.32.5 closed by remote host.
Connection to 172.20.32.5 closed.
Transferred: sent 2992, received 47976 bytes, in 94.1 seconds
Bytes per second: sent 31.8, received 510.1
debug1: Exit status -1
Whenever a connection dies to any of my servers its aborted with closed by remote host message.
Even SSH ZFS replications on TrueNAS, to a different subnet, loose their SSH connection the same way.
When SSHing inside of one subnet, so not going through the OPNsense appliance it works flawlessly.
Also another example. When trying to iperf3 to a different subnet, there is simply starvation. No traffic at all. Going through the directly connected subnet it works fine.
chairman@fedora:~$ iperf3 -c 192.168.200.5
Connecting to host 192.168.200.5, port 5201
[ 5] local 192.168.3.22 port 50422 connected to 192.168.200.5 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 386 KBytes 3.16 Mbits/sec 2 1.41 KBytes
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 386 KBytes 316 Kbits/sec 5 sender
[ 5] 0.00-10.00 sec 65.0 KBytes 53.3 Kbits/sec receiver
iperf Done.
chairman@fedora:~$ iperf3 -c 192.168.3.5
Connecting to host 192.168.3.5, port 5201
[ 5] local 192.168.3.22 port 41500 connected to 192.168.3.5 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 946 Mbits/sec 12 153 KBytes
[ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec 9 158 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 939 Mbits/sec 6 212 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 11 96.2 KBytes
[ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec 12 204 KBytes
[ 5] 5.00-6.00 sec 111 MBytes 932 Mbits/sec 9 136 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 940 Mbits/sec 10 161 KBytes
[ 5] 7.00-8.00 sec 111 MBytes 932 Mbits/sec 11 168 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 932 Mbits/sec 13 171 KBytes
[ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 9 184 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.09 GBytes 936 Mbits/sec 102 sender
[ 5] 0.00-10.00 sec 1.09 GBytes 934 Mbits/sec receiver
iperf Done.
What is wrong here? Did anyone encounter some similar behaviour before?
Thanks in advance!