Unable to SSH between Subnets

[user88]

I recently set up my own Opnsense server at home, and everything has been going great except for a couple small issues. The main one is that I cannot SSH from a device on one subnet to a device on another subnet. Before I explain the issue in detail, I will explain my current network setup.

My network is as follows:

- Internet comes into house to ISP router (subnet 192.168.0.1/24).
- Opnsense box
  - WAN: Attached to ISP router (address 192.168.0.10)
  - LAN1: Goes to switch for wired network (subnet 192.168.10.1/24).
  - LAN2: Goes to wireless AP for wireless network (subnet 192.168.11.1/24).

I have a box with SSH open on the LAN1 subnet, and I realized I could not SSH into it from my laptop on the LAN2 network (wireless). The specific error shown is "Connection closed by 192.168.10.x port 22". Now if I instead SSH from a machine on the LAN1 subnet (same as the SSH server) it works with no issue. I assumed at first that a firewall rule was blocking it, but I have confirmed from the logs that the firewall is passing the traffic correctly.

To make it more interesting, I observe similar behaviour when I try and SSH from my laptop (LAN2) to my cloud server which has a public IP address. If I try and SSH from LAN2, I get the same "Connection closed" error, but if I instead connect to the wireless hotspot on my phone and SSH again, there is no issue.

Because of these symptoms, I am inclined to believe this is a NAT issue, but I am confused for a couple reasons:

Reason 1: There is connectivity between subnets. I can ping between subnets no problem, and I even setup a netcat tunnel on the SSH port between the machines on separate subnets without issue.

Reason 2: My previous network setup had my AP in router mode with the ISP router plugged into it's WAN port, so I did not have any separate subnets like I do now, but NAT wise it was pretty much the same setup as far as the SSH connection to my cloud server goes, which did work on that setup.

In my NAT configuration, I have a the following rules:

- Interface: WAN, Source: Loopback net, Destination: *, NAT address: Interface address.
- Interface: WAN, Source: LAN net, Destination: *, NAT address: Interface address.
- Interface: WAN, Source: LAN2 net, Destination: *, NAT address: Interface address.
- Interface: LAN, Source: LAN net, Destination: LAN2 net, NAT address: Interface address.
- Interface: LAN2, Source: LAN2 net, Destination: LAN net, NAT address: Interface address.

Am I missing something here? What stumps me is that if I can successfully create a netcat tunnel between the two devices, why would SSH not work? I am relatively new to firewalls and opnsense so any help is appreciated. Thanks.

[meyergru]

I would not create NAT rules for LAN1 and LAN2, but instead create firewall rules and set up routing. The way you are doing it will probably confuse the machines, because the IPs are NATed both ways, which cannot work.

Outbound NAT is only for WAN interfaces where you have to "hide" behind the one routeable IP your ISP gives you. The direction LAN -> LAN2 should work right from the start, because in default, there is an "allow all" rule in place for the first LAN. For any new LAN, you must create it yourself or be more specific in what you want to allow.

[user88]

@meyergru Thanks for great explanation. I see what you are saying. Now that I think about it, you're right I shouldn't need NAT between the two interfaces. I have now removed those NAT rules. I had already set up a LAN2 to any firewall rule to match the auto generated LAN rule of the same kind. They are as follows:

 - Interface: LAN, Protocol: IPv4 *, Source: LAN net, Destination: *.
 - Interface: LAN2, Protocol: IPv4 *, Source: LAN2 net, Destination: *.

That should be sufficient right? I can even see the firewall passing this traffic in the logs but its still not working. To me (someone inexperienced with networking) this seems like it should be working because I can't find the root of the issue. My initial assumption is that something on the SSH configuration was not set up correctly, but I have SSH configs for these hosts that worked no problem before I setup this firewall, and SSH from the same subnet works without issue, its only across subnets that is problematic. So to me it seems like it has to be something firewall related. I am interested to know your thoughts.

[meyergru]

If you can access the ports with tools like netcat or nmap, you should be good network-wise. But also, there might be some other local firewall rules in place from things like ufw, network restrictions that only allow connections from your local subnets or maybe wrong ciphers when the source and destination disagree on what ciphers are allowed. The latter would apply only to SSH traffic, obviously.

[EricPerl]

Are you still getting this connection closed error or does ssh just hang on the client side?

If you had enabled logging on your IN rules on the LAN2 interfaces, you would see an entry in the log first.
IN on LAN2 (from the client), OUT on LAN (towards the server).

Observing reply traffic is a little more difficult but not that hard with SSH.
Head to interfaces > diagnostic > Packet capture and filter on port 22.
You should observe the packets triggering the log entries above, then the replies back (from the server on LAN, to the client on LAN2)...

If you don't see a reply hitting LAN (from the server), you'll have to dig deeper on that server.

[user88]

@meyergru I agree that it should be fine network wise at this point. No firewalls are setup on the servers, and if I connect my laptop to the LAN network instead (via wired ethernet) I can ssh no problem. It's certainly stumping me.

@EricPerl Yes I am still getting the connection closed error. Specifically "Connection closed by 192.168.10.91 port 22" in which 192.168.10.91 is the SSH server on the LAN network. I ran the diagnostics like you recommended and as far as I can tell it seems like the reply is getting back from LAN to LAN2. I've attached images of the diagnostic run for you to look at. One is for the LAN network and the other is for LAN2 (OPT1). It seems they are sorted by interface name rather than timestamp but looking at the timestamps seems to show a reply being sent. I am interested to know your thoughts. Thanks.

[passeri]

@user88 please try and report on ssh -vvv <login>

It looks to me like Opnsense is fine, by your report, while the message is typical of incorrect credentials although it may be something else.

[user88]

Hi @passeri, thanks for the reply. I have attached an image of the ssh -vvv output you requested. By my own analysis, it seems that it's failing in the key exchange identification function. I was able to find a related post on serverfault in which the chosen answer describes the root cause of the error message. So it seems that based on this error, the client side determined that there was no process listening on the other side of the connection, whether that's because the server closed it or something else did.

What's odd is that the auth logs at "/var/log/auth.log" on the SSH server show "Connection closed by <client ip address> port 52958 [preath]" when I try to SSH from the client. So it seems each side is saying that the other closed the connection. Obviously both can't be true, so either the error wording is misleading or maybe something else is closing the connection? This would be far less puzzling if I wasn't able to SSH into this server easily from the LAN subnet. Is there any other process that could be interfering with this? I do have zenarmor running but I don't see how that could interfere with it.

[passeri]

I believe you have no matching key. Observe you have "type -1" after each test of a key or certificate type. If I run the same command to SSH into my server on a different subnet through Opnsense I get a "type 0" for id_rsa and login succeeds at the point where your connection fails.

You will see in the link you provided, where there is successful connection and subsequent failure from another cause, that the first listing has "/id_rsa type 0" hence it found a matching key and could continue. -1 is the standard POSIX error code.

The fact you can login from another subnet is a distraction. It offers no guarantee that you have keys set up correctly on the machine on the second subnet. I would check all of those carefully, or try moving the working machine across.

I am making a best estimate here from the information. There are other possibilities. My "not a guru" status is unthreatened. :-)

[user88]

The thing is I am not using an SSH key as authentication. This SSH server uses password auth. When I say I can successfully login from another subnet, I mean that when I take my laptop which is unable to SSH from subnet 1 and turn off the wifi and connect it to subnet 2 (via ethernet), I can successfully SSH into the SSH server that resides on subnet 2.

Looking at the -vvv output for SSH on this successful connection from subnet 2, I see that the type is also showing as -1 (error) because each of the keys it's checking are for other servers, and this SSH server uses password auth, which it eventually defaults to when I SSH from subnet 2, but for some reason it never reaches this stage when initiating from subnet 1.

To make things more interesting, I just tried to push some changes to my github over SSH and I get the following error:

Code Select Expand

Connection closed by 140.11.121.3 port 22
fatal: Could not read from remote repository.

This issue persists whether I am connected to subnet 1 or subnet 2. But if I instead connect to the wireless network from my ISPs router, bypassing my opnsense firewall, I can push over SSH with no problem, and SSH to my cloud server as well with no issue. So something has to be wrong with my opnsense setup. If its not firewall rules and its not the NAT configuration, what could it be?

[passeri]

My erroneous assumption about use of keys, sorry.

I will go back to read again. The new test about accessing Github is interesting although the difference in error message also creates some uncertainty.

Given I and others cross subnets using SSH through Opnsense without a problem, there should be a comparison, test or review we can make somewhere to locate your issue.

[meyergru]

I know that you can forbid password authentication for SSH for specific users or root. In the latter case, you would get that message if you did not provide a key.

You can also forbid password authentication for different subnets, for example with something like "Host 192.168.10.*", see ssh_config(5) manpage. I would suggest looking at the SSHD configuration on the host. The specific setting can be hidden in different places, like e.g. /etc/ssh/sshd_config, /etc/ssh/sshd.d/*.conf, ~user/.ssh/....

Guessing from the "fatal: Could not read from remote repository." message, your SSH host is a local Git repository, so I bet the configuration is more complex than you might think.

FWIW, I bet this is not a network problem any more.