[SOLVED] Unable to SSH between Subnets

[Patrick M. Hausen]

I did not follow all the details (sorry) but this smells like elves ... er ... like some overlooked NAT rule that leads to the SSH connection going to a different server to me.

[EricPerl]

Is it possible that your client is now jailed by the server because it didn't complete auth?
https://www.openssh.com/txt/release-9.8
Check out the new features section.
Of course, I don't know that the server is 9.8...

[user88]

It does feel like something NAT related, but then again I can create a netcat tunnel on port 22 no problem and send data back and forth...

OpenSSH version is 9.6p1, but I don't think it could be that anyways because I can SSH in successfully by just moving my laptop from the wireless network (LAN2) to the wired network (LAN1). To me, that removes all possibility for issues with the server or client config and leaves something else in the middle. What that is, I don't know yet. Everything seems to be checking out except it's still not working...

[EricPerl]

The packet captures didn't indicate anything like NAT issues. IP:port was equal on both interfaces.
The only thing worth checking there was that MAC addresses matched NICs involved.

If your server is 9.6p1 (like mine), then it can't be that. Your client showed 9.8 so I thought it might apply to your server as well.
Apparently, for that feature, the penalty is tracking clients by IP/subnet so changing network would be sufficient to clear that hurdle.
I don't know how early the connection is dropped.

[dseven]

MTU issue? As a quick test, try lowering the MTU on the client manually and see if SSH works...

[cookiemonster]

libressl incompat perhaps? I suggest put another ssh machine without libressl and instead openssl on the same network segment and try again.
Debug info seems to suggest not a networking problem due to the successful connection messages and unsuccessful ones relating to not matching keys. The successful connection to it from within the same segment tells different, I get that. An interesting one. Diagnosing application nevertheless. The usual elimination process applies ;)

[meyergru]

Do you have a port forward from WAN to port 22 that could be misconfigured and catches your SSH connections?

[user88]

No luck with the MTU unfortunately. I have no port forward on the WAN for port 22.

@cookiemonster What part of the setup concerns you with libressl? My limited knowledge seems to recall that it's an OpenBSD fork of OpenSSL. My laptop and server both use OpenSSL as far as I know. Maybe I'm missing something?

[passeri]

User88, I re-read your opening post. What happens if you disconnect the AP from Opnsense, connecting your computer directly to that interface? I did not spot that you had tried that.

Also, is your OPNSense default apart from configuring interface IPs and a single firewall rule allowing LAN2 net to Any? On your description nothing more should be required for operation.

[dseven]

Sorry, TL;DR, but... have we confirmed that the ssh connection is actually making it to the intended destination host? Could do this with either a packet capture (on the ssh server itself), or maybe ssh server logs? I'm leaning towards some sort of NAT redirect now too....

[EricPerl]

My read of reply #5 (per my request) was that some traffic was flowing forth and back through both interfaces involved, without any NAT or PF.
I suggested to check if the MAC addresses matched systems involved. Not confirmed yet.

Reply #7 has a Connection established line (right before the big white block) that matches. IME, you don't get there with broken connectivity.
It somehow seems to fail early during key exchange, not even displaying the remote version.

Apparently, the same systems can complete the connection when on the same LAN.
IMO, that eliminates issues around incompatible clients.

I thought I was onto something when I noticed 9.8 and the new fail2ban feature but the server is 9.6p1...
I'm out of ideas.

[passeri]

Given it is known to work on the same subnet, I proposed above to simplify by eliminating the wireless access point and configure the most basic working Opnsense. That will eliminate doubt or distractions around the actual problem. In their recent post user88 was still referring to a potential NAT problem, which (if it or anything similar exists) must lie in configuration or in the wAP.

[user88]

@dseven Yes, we have confirmed that. I did a packet capture (shown on the previous page of this thread) and it does reach the destination server. @EricPerl I just went to double check and yes the MAC addresses all match up with what they should be so that must not be the issue.

@passeri I will try disconnecting the AP and just connecting my laptop tonight. Sorry for the late replies, have been busy. Will update soon.

[user88]

I just tried connecting my laptop directly to the LAN2 port of my opnsense box. SSH still disconnects the same as before when I try to connect to my server on LAN1. There is something that is wrong with the whole opnsense here in my opinion, because not only is it doing this across subnets, but I cannot use SSH to push code to github.com either. It fails with the same symptoms. So basically if the SSH connection goes through an interface then it doesn't work.

[EricPerl]

As I said, I'm out of ideas as to what's out of whack with your entire setup, but SSH is in fact the first use case I typically try when troubleshooting.
Port forwarding, inter-VLAN, VPN home... On Ubuntu, Windows, a bunch of networking devices and so on.

I've had issues with incompatible crypto and password availability. Nothing like your issue.