I've recently migrated a firewall from pfSense to opnSense. However, I've been experiencing issues with filesharing even though the same firewall rules are in place. The connection seems to occur, but I get a "system error 53" and the connection resets when attempting to connect to the fileshare.
The issue seems to occur only when the client computer that is connected to a domain attempts to connect to a server which is not domained.
I basically have a virtual firewall that has segmented a web server outside our domain (DMZ). We make updates to the server via SMB/file explorer.
Computers that are NOT on a domain can connect to the shares just fine. A prompt for user/pass is shown and a connection is made. Computers that ARE on a domain just throw a system 53 error. This doesn't occur on the pfSense box with basically the same configuration. Everything is default except the firewall rules and port forwarding to the web server ip.
I have provided screens of the firewall rules here https://imgur.com/a/pDbkGP6.
All other protocols seem to work fine except SMB. HTTP/HTTPS/DNS/SSH/ORACLE/MYSQL all connect and authenticate successfully.
Few things I tried:
- Ensure local group policy is set to "Digitally sign communications (always)" to match the group policy setting of the domain https://blogs.technet.microsoft.com/josebda/2010/12/01/the-basics-of-smb-signing-covering-both-smb1-and-smb2/"
- Allow everything
I have restored the pfsense vm for now and file shares are working again, but it seems strange that the same settings dont work on opnsense.
I have a few other opnsense machines that are working fine, but they aren't operating under a scenario similar to this.
Any ideas?
It's worth noting that there is nothing logged about blocking the connection to the web server when attempting to connect that I can see.
I noticed that "Windows Filtering Platform" was not enabled on the virtual switch for OpnSense, so I enabled that, but that had no effect.