Cant SSH to remote site on alt port after 17.1.1

[RickNY]

After updating my 17.1 machine to 17.1.1 this afternoon, I can no longer initiate an outgoing SSH connection to a remote server that listens for SSH connections on port 922.. If I try to perform the following from a shell connected directly to the router:

 ssh remote-host.com -p 922 -l myusername

I get:
ssh: connect to host remote-host.com port 922: Operation not permitted

My LAN clients that attempt to connect to same timeout now.  I have a default outgoing LAN firewall rule that allows outgoing connections to all, and there is nothing being reported in the firewall logs that it is being blocked.

For the time being, I have changed the remote SSH server to listen on port 22, and that does work.  Likewise, if I attempt to change it to port 9222, it works. 

Did something change

[bartjsmit]

Do you have Suricata configured?

Bart...

[RickNY]

No, Suricata is not configured. It's a pretty vanilla setup with a few port forwarding rules, and a default outgoing LAN rule that allows all except for external DNS.

[bartjsmit]

I've added 922 to sshd on a host here and I'm able to connect from the shell of my test OPNsense 17.1.1 VM just fine with ssh user@<host IP> -p 922

There doesn't seem to be an intrinsic problem with 17.1.1 that is causing this.

Bart...

[RickNY]

My thinking is that something got out of sorts during the upgrade process... Also, FWIW -- the admin SSH port on the firewall is also set to 922... I tried temporarily changing that, but 922 is still not making it out.

The one remote possibility is that Digital Ocean (my VPS provider) is blocking 922 from that specific IP address -- but I'm highly doubtful of that -- although I've opened up a ticket with them to confirm.

[RickNY]

Additional information... Doing a packet capture on the WAN interface, the packets are not making it out of the WAN interface when attempting port 922.  If I change the port number to anything else, the outgoing attempt shows up in the capture.

[RickNY]

Hmmm... I think I found something.. My system isn't running 17.1.... When I did the update from the web interface yesterday, it apparently updated to 17.7.a_28-amd64

How did that happen?

Edit: I just ran a  opnsense-update -t opnsense  -- and that brought it back to 17.1.1... Not sure why it pulled the development version.. The SSH outgoing to port 922 is still not working.. FWIW -- attempting to SSH anywhere with port 922 is throwing the "Operation not permitted" error on the command line.

[bartjsmit]

Did you reboot after changing the SSH port back to 22?

Bart...

[RickNY]

Yes, its been rebooted since changing it.

Attempting to any host, with just that port -- fails:

root@firewall:~ # ssh -p 923 1.1.1.1 -vvvv
OpenSSH_7.4p1, OpenSSL 1.0.2k  26 Jan 2017
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug2: resolving "1.1.1.1" port 923
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 1.1.1.1 [1.1.1.1] port 923.
^C
root@firewall:~ # ssh -p 922 1.1.1.1 -vvvv
OpenSSH_7.4p1, OpenSSL 1.0.2k  26 Jan 2017
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug2: resolving "1.1.1.1" port 922
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 1.1.1.1 [1.1.1.1] port 922.
debug1: connect to address 1.1.1.1 port 922: Operation not permitted
ssh: connect to host 1.1.1.1 port 922: Operation not permitted

Any other port -- works.

[RickNY]

OK.. Found it -- but not sure why it manifested itself as an issue just now.

I have 3 floating rules set up to allow incoming SSH access to the router.. The 1st two rules are allow rules, allowing admin SSH connections from two specific IP addresses.  The 3rd rule was a block.. The direction on that was set to "any".  This had always been the case -- and it was not intefering with outgoing NAT connections to port 922 -- until I updated to 17.1.1.

SO, changing the direction of the rule to "in" resolved this...  So -- for curiosity sake -- what changed in 17.1.1 that would have suddenly caused this rule to be processed properly.  My guess is the order of NAT operations and when the filter rules get applied?

Thanks,
Rick

[djGrrr]

17.1.1 fixed the "any" rules:

firewall: direction "any" was not respected in floating rules

[RickNY]

Ah... So my broken rule hasn't been working all this time.  Glad its been fixed.