Messages

Ya, puzzles me too. Definitely should have just disabled it. Thanks again for the help. Marked as solved!

Alright, after carefully going through my full configuration of Opnsense it turns out that Zenarmor was the issue. There was a policy configured under "App Controls -> Remote Access" which was blocking SSH. Sure enough I turned it off and everything was fine. A wild goose chase that should never have happened... Thanks to everyone here for all their help and to @passeri for suggesting I just try with a default Opnsense!

If there is a way to mark this as solved, please let me know and I will do that.

I just tried connecting my laptop directly to the LAN2 port of my opnsense box. SSH still disconnects the same as before when I try to connect to my server on LAN1. There is something that is wrong with the whole opnsense here in my opinion, because not only is it doing this across subnets, but I cannot use SSH to push code to github.com either. It fails with the same symptoms. So basically if the SSH connection goes through an interface then it doesn't work.

@dseven Yes, we have confirmed that. I did a packet capture (shown on the previous page of this thread) and it does reach the destination server. @EricPerl I just went to double check and yes the MAC addresses all match up with what they should be so that must not be the issue.

@passeri I will try disconnecting the AP and just connecting my laptop tonight. Sorry for the late replies, have been busy. Will update soon.

No luck with the MTU unfortunately. I have no port forward on the WAN for port 22.

@cookiemonster What part of the setup concerns you with libressl? My limited knowledge seems to recall that it's an OpenBSD fork of OpenSSL. My laptop and server both use OpenSSL as far as I know. Maybe I'm missing something?

It does feel like something NAT related, but then again I can create a netcat tunnel on port 22 no problem and send data back and forth...

OpenSSH version is 9.6p1, but I don't think it could be that anyways because I can SSH in successfully by just moving my laptop from the wireless network (LAN2) to the wired network (LAN1). To me, that removes all possibility for issues with the server or client config and leaves something else in the middle. What that is, I don't know yet. Everything seems to be checking out except it's still not working...

I agree with your sentiment but I am having a really hard time deciphering exactly what else could be the problem. The ssh configuration on this server is extremely basic. The only uncommented lines of the configuration are as follows:

Code Select Expand

Include /etc/ssh/sshd_config.d/*.conf
Port 22
PasswordAuthentication yes
KbdInteractiveAuthentication no
UsePAM yes
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_*
Subsystem       sftp    /usr/lib/openssh/sftp-server
In fact the only thing I changed after the fresh install of ubuntu on the SSH config was uncommenting "Port 22" and "PasswordAuthentication yes".

@meyergru The github host is not local, it is the actual github.com server. I certainly was able to push to it before installing my opnsense firewall, and nothing else has changed on my network except the addition of the firewall. To push my changes, I could not even push from the LAN network after connecting my laptop to it, I had to connect to my ISP access point and I was able to push with no issues.

I am no network expert but it seems like the issue is only presenting itself when the destination is outside the current subnet, which keeps making me think its a NAT issue. I plan to set up a fresh ssh box on my LAN2 network and try to SSH from the LAN network. If that doesn't work, I don't see how it could be anything other than a network problem.

The thing is I am not using an SSH key as authentication. This SSH server uses password auth. When I say I can successfully login from another subnet, I mean that when I take my laptop which is unable to SSH from subnet 1 and turn off the wifi and connect it to subnet 2 (via ethernet), I can successfully SSH into the SSH server that resides on subnet 2.

Looking at the -vvv output for SSH on this successful connection from subnet 2, I see that the type is also showing as -1 (error) because each of the keys it's checking are for other servers, and this SSH server uses password auth, which it eventually defaults to when I SSH from subnet 2, but for some reason it never reaches this stage when initiating from subnet 1.

To make things more interesting, I just tried to push some changes to my github over SSH and I get the following error:

Code Select Expand

Connection closed by 140.11.121.3 port 22
fatal: Could not read from remote repository.

This issue persists whether I am connected to subnet 1 or subnet 2. But if I instead connect to the wireless network from my ISPs router, bypassing my opnsense firewall, I can push over SSH with no problem, and SSH to my cloud server as well with no issue. So something has to be wrong with my opnsense setup. If its not firewall rules and its not the NAT configuration, what could it be?

Hi @passeri, thanks for the reply. I have attached an image of the ssh -vvv output you requested. By my own analysis, it seems that it's failing in the key exchange identification function. I was able to find a related post on serverfault in which the chosen answer describes the root cause of the error message. So it seems that based on this error, the client side determined that there was no process listening on the other side of the connection, whether that's because the server closed it or something else did.

What's odd is that the auth logs at "/var/log/auth.log" on the SSH server show "Connection closed by <client ip address> port 52958 [preath]" when I try to SSH from the client. So it seems each side is saying that the other closed the connection. Obviously both can't be true, so either the error wording is misleading or maybe something else is closing the connection? This would be far less puzzling if I wasn't able to SSH into this server easily from the LAN subnet. Is there any other process that could be interfering with this? I do have zenarmor running but I don't see how that could interfere with it.

@meyergru I agree that it should be fine network wise at this point. No firewalls are setup on the servers, and if I connect my laptop to the LAN network instead (via wired ethernet) I can ssh no problem. It's certainly stumping me.

@EricPerl Yes I am still getting the connection closed error. Specifically "Connection closed by 192.168.10.91 port 22" in which 192.168.10.91 is the SSH server on the LAN network. I ran the diagnostics like you recommended and as far as I can tell it seems like the reply is getting back from LAN to LAN2. I've attached images of the diagnostic run for you to look at. One is for the LAN network and the other is for LAN2 (OPT1). It seems they are sorted by interface name rather than timestamp but looking at the timestamps seems to show a reply being sent. I am interested to know your thoughts. Thanks.

@meyergru Thanks for great explanation. I see what you are saying. Now that I think about it, you're right I shouldn't need NAT between the two interfaces. I have now removed those NAT rules. I had already set up a LAN2 to any firewall rule to match the auto generated LAN rule of the same kind. They are as follows:

 - Interface: LAN, Protocol: IPv4 *, Source: LAN net, Destination: *.
 - Interface: LAN2, Protocol: IPv4 *, Source: LAN2 net, Destination: *.

That should be sufficient right? I can even see the firewall passing this traffic in the logs but its still not working. To me (someone inexperienced with networking) this seems like it should be working because I can't find the root of the issue. My initial assumption is that something on the SSH configuration was not set up correctly, but I have SSH configs for these hosts that worked no problem before I setup this firewall, and SSH from the same subnet works without issue, its only across subnets that is problematic. So to me it seems like it has to be something firewall related. I am interested to know your thoughts.

UPDATE: The issue was a Zenarmor policy that was blocking SSH connections.

I recently set up my own Opnsense server at home, and everything has been going great except for a couple small issues. The main one is that I cannot SSH from a device on one subnet to a device on another subnet. Before I explain the issue in detail, I will explain my current network setup.

My network is as follows:

- Internet comes into house to ISP router (subnet 192.168.0.1/24).
- Opnsense box
  - WAN: Attached to ISP router (address 192.168.0.10)
  - LAN1: Goes to switch for wired network (subnet 192.168.10.1/24).
  - LAN2: Goes to wireless AP for wireless network (subnet 192.168.11.1/24).

I have a box with SSH open on the LAN1 subnet, and I realized I could not SSH into it from my laptop on the LAN2 network (wireless). The specific error shown is "Connection closed by 192.168.10.x port 22". Now if I instead SSH from a machine on the LAN1 subnet (same as the SSH server) it works with no issue. I assumed at first that a firewall rule was blocking it, but I have confirmed from the logs that the firewall is passing the traffic correctly.

To make it more interesting, I observe similar behaviour when I try and SSH from my laptop (LAN2) to my cloud server which has a public IP address. If I try and SSH from LAN2, I get the same "Connection closed" error, but if I instead connect to the wireless hotspot on my phone and SSH again, there is no issue.

Because of these symptoms, I am inclined to believe this is a NAT issue, but I am confused for a couple reasons:

Reason 1: There is connectivity between subnets. I can ping between subnets no problem, and I even setup a netcat tunnel on the SSH port between the machines on separate subnets without issue.

Reason 2: My previous network setup had my AP in router mode with the ISP router plugged into it's WAN port, so I did not have any separate subnets like I do now, but NAT wise it was pretty much the same setup as far as the SSH connection to my cloud server goes, which did work on that setup.

In my NAT configuration, I have a the following rules:

- Interface: WAN, Source: Loopback net, Destination: *, NAT address: Interface address.
- Interface: WAN, Source: LAN net, Destination: *, NAT address: Interface address.
- Interface: WAN, Source: LAN2 net, Destination: *, NAT address: Interface address.
- Interface: LAN, Source: LAN net, Destination: LAN2 net, NAT address: Interface address.
- Interface: LAN2, Source: LAN2 net, Destination: LAN net, NAT address: Interface address.

Am I missing something here? What stumps me is that if I can successfully create a netcat tunnel between the two devices, why would SSH not work? I am relatively new to firewalls and opnsense so any help is appreciated. Thanks.