[SOLVED] Unable to SSH between Subnets

[passeri]

@user88, that clarifies to a point. For the other half, have you tried with a fresh, purely default Opnsense?

[EricPerl]

Hmm, after additional research, I'm no longer sure the "Connection established" log indicates a successful roundtrip.

Couple more ideas:
1. Check /etc/hosts.allow & /etc/hosts.deny
2. Start a second ssh deamon with full debug on an alternate port, eg: sshd -ddd -p 2222 (you may need to modify your FW rules. You also have to specify that port on the client side). The server debug output may reveal interesting data.
3. The packet captures were promising. We only looked at the existence of packets though. The first few may actually be clear since key exchange has not occurred yet. Can you get the content? It would be nice to see the remote version in the high details output.

For example:
You cannot view this attachment.

Interestingly, when using the Windows client, the remote string is sent first. The WSL client does this the other way around.
I saw something along these lines in the code, but I have not looked for the trigger for that behavior.
IMO, seeing a packet with server issued content would exonerate OPN.
Such capture is best done on the interface of the client.

[user88]

Alright, after carefully going through my full configuration of Opnsense it turns out that Zenarmor was the issue. There was a policy configured under "App Controls -> Remote Access" which was blocking SSH. Sure enough I turned it off and everything was fine. A wild goose chase that should never have happened... Thanks to everyone here for all their help and to @passeri for suggesting I just try with a default Opnsense!

If there is a way to mark this as solved, please let me know and I will do that.

[EricPerl]

Geez... How the hell does this align with the packet captures showing traffic????

I'm aware of the order in which Zenarmor and OPN handle traffic, but for traffic to be visible in OPN and blocked by Zenarmor, packets have to be dropped on the way out... which is pretty weird.

Anyway, I never saw the Zenarmor mention.
It's also the first thing I would have disabled in this scenario.
Overlapping layers of security just make troubleshooting a pain.

[EricPerl]

Oh, just edit the subject of the OP.

[user88]

Ya, puzzles me too. Definitely should have just disabled it. Thanks again for the help. Marked as solved!