FTP proxy data channel cant get through

[pelle]

This traffic 'drawing' from both side of the OPNsense/FTP proxy plugin, indicate that it fails to 'prepare' for the FTP data channel (if you ask me). It handles the FTP control channel as expected.

        INTERNET                                FTP DMZ
ExtClient(InetIP)-(DHCP-DynDNS)OPNsense(10.1.1.1)-(10.1.1.2)FTP Srv(vsftpd)
                                 with
                               FTPproxy
                                plugin

  TCP port 21 setup and FTP ->!        ! TCP port 21 setup and FTP ->
  login works fine every time !        ! login works fine every time

                          Then this happens

         FEATreq.dstport21--> !        !
                              !        ! FEATreg.1->.2.dstport21--->
                              !        !<--EPRT,EPSV,MDTM,PASV,
                              !        !REST S.,SIZE,TVFS,UTF8
        <-EPRT,EPSV,MDTM,PASV,!        !
        REST S.,SIZE,TVFS,UTF8!        !
        EPRT-1-InetIP-55838-->!        !
                              !        ! EPRT-1-.1-54376--->
                              !        !<--EPRT-Succ.Consid.EPSV
      <--EPRT-Succ.Consid.EPSV!        !
                       LIST-->!        !
                              !        ! LIST-->
                              !        !
            -------           !    -   !<-SYN.1.dst54376.2.src20
                              !    !   !<-SYN.1.dst54376.2.src20 (Retransm. 1s)
            Nothing           !    !   !<-SYN.1.dst54376.2.src20 (Retransm. 2s)
            happens           !   30s  !<-SYN.1.dst54376.2.src20 (Retransm. 4s)
             here!            !    !   !<-SYN.1.dst54376.2.src20 (Retransm. 8s)
                              !    !   !<-SYN.1.dst54376.2.src20 (Retransm. 16s)
            -------           !    -   ! ICMP.1 host .1 unreach. -> ?From .1?
                              !        !
                              !        !<--Src.2.21 FTP 425 Failed conn
                              !        ! TCP ack from .1 --> (?Why ICMP unreach?)
     <--Src.21 425 Failed conn!        !


It does not matter if I do it with passive or active FTP. The data session does not come through. Still, the control channel works both with passive and active. As you can see from my 'drawing', the FTPproxy does its job for the control channel. It does change the IP src/dst in both directions, and even change the IP inside the EPRT FTP command as it should. It's all about the FTP proxy fail to prepare to open the 'data channel' port in the firewall together with a 'temporary NAT' (= my guess).

At least, this is how I guess an FTP proxy should work. Maybe I have missed something, but I have tried all FTP proxy settings I can come up with, like source IP address, and rewrite source port. I have moved the 'FTP rules/NATs' to the top in each list, I have created a 'dummy' rule to allow all traffic out from the FTP server. It still can't handle the dynamic data channel TCP port session.

I also run another FTP proxy on another OPNsense. Its a 'client-based' configuration, no reverse address/port, which works fine when I connect to public FTP servers. My problem starts when I want to have an FTP server (vsftpd) on the inside (FTP DMZ) of an OPNsense setup. I have tried to remove and reinstall the FTP proxy plugin. I have rebooted before, between and after all my tests. Still the same result.

I have tcpdump in the FTP server, packet capture in OPNsense and Wireshark on the outside to understand where the FTP fails. Maybe there are some log's I have not found, which might tell me something more useful. But as a non-Linux guy, I have so far missed such log's

One more thing, the OPNsense have ten inside interfaces (opt's), one for each DMZ servers it handles. It also has an internal interconnect link (lan) for management . . . and of course the WAN outside Inet interface (wan). All interfaces are 'virtual' created as Proxmox VLAN's (= OPNsense does not know each interface is a VLAN tag in the Proxmox guest setup). From the OPNsense perspective, it has 12 real interfaces (vtnet 0-11). I can't see why this should have anything to do with the FTP data channel, but a computer is a computer, a sensible machine with its own will So better to tell you the setup right away. I have tested this setup from two different FTP client software's, from an iPad and a PC. Both fail to connect the FTP data channel, even if the control channel works fine.

FTP proxy is set up as mention in the OPNsense documentation. A 127.0.0.1/8021 proxy with a NAT 'catch' on the WAN interface with a FW rule allow port 21 in on the WAN. And as you can see, it seems to work fine as long as it only handles FTP control traffic (port 21).

So, any suggestions? Should my multi-interface FTP proxy setup work or should I go in some other direction? What should be my next step? Maybe I can use NAT only in some way, without the FTP proxy plugin, by allowing a narrow TCP port range to be NAT'ed to the vsftpd DMZ server. Any suggestions would kindly be accepted. I cant come up with more things to investigate, and I can't figure it out. If I can't get any good advice from this query, I have to start looking somewhere else then OPNsense . . . and I don't want to leave OPNsense. It has so much to offer and drop it because of such a 'small' thing as an FTP problem hurts.

Best Regards and thanks for all help I can get.
- Per Håkansson

[fabian]

Check your firewall rules, you may block the data connection.

[pelle]

Thanks, Fabian

Made me thinking. Rules for data channel too?? I believed that was the job for FTP proxy, to temporarily create data channel rules from what's in the control channel. But after your comment, I did realise I maybe had 'hoped' too much of the FTP proxy.

So, after once again clean out my OPNsense from everything regarding FTP proxy, I reinstall the FTP proxy again, add NAT to 127.0.0.1 and add a port 21 rule on the WAN as stated in the documentation.

This time, I also added an "any any" rule on my FTP DMZ interface letting the FTP server do whatever it likes (I don't like that, but what the heck, it's a test). I also add a rule on the WAN interface allowing specific vsftpd passive TCP destination ports to pass to the FTP server. This WAN rule was something new to me (and I'm not too fond of it from a security point, even if there are no inbound NAT so that would be pretty safe anyway I think).

And now passive FTP works, yeah great!!! I now have understood that the FTP proxy adds a temporarily NAT for the data channel to the FTP server. It does *not* add a firewall rule on the other hand. This 'auto-NAT but not auto-rule' function is useful to know.

When I try to use active FTP, my firewall log show that the FTP data channel is allowed from my FTP server out to the external FTP client, inside-ftp-serv:20 -> extIP.23456 due to my "any any" rule. But it still not working. I will have to do the same packet capture for this problem. I suspect that the data channel port number is not translated correctly from inside to outside (but its a guess and I have to verify it). I will add whatever I find out to this post when I have captured the traffic.

Once again Fabian, thanks for 'open my eye' on how the FTP proxy works. I am a network guy mostly working with Cisco ASA, and I build my assumption from it's FTP inspection. Sorry for not 'thinking it through'. I hope my problem with active FTP also depends on 'wrong thinking'

Best Regards
- Per Håkansson

[fabian]

I thought more about some things like a custom deny any rule at the bottom which breaks FTP quite frequently for many users.

[pelle]

Hi, Fabian

You were right this time too. It was a rule denying the active data channel creation, but not in my DMZ OPNsense. This time, the data channel was blocked on the client-side FW. Thats why I didn't see any block lines in the DMZ FW log.

But, I will not open the client-side OPNsense to all incoming high TCP port just to let active FTP works. I rather make active FTP fail. I know that the FW still needs a NAT/ALG/PROXY function to 'convert' the outside request into an inside connection. But somehow, this "allow-all-high-TCP-ports" open ups the FW for anyone who wants to try . . . or am I foolish and have misunderstood the whole concept of an FW? I usually don't even answer ICMP pings or sending any other ICMP message on my WAN.

My conclusion: I have an FTP server-side proxy function allowing both passive and active FTP sessions, that's good. I will not open up the WAN side of my client OPNsense for high order TCP ports to support active FTP.

Is there any discussion having FTP proxy create temporarily rule's for the data channel? The proxy software has all the information it needs looking at the control channel to make such a temp rule in the firewall rule-set. That would be a very 'smooth' feature . . . if you ask me (even if this topic tells you I'm not the best to figure stuff out

Once again, Fabian, thanks a lot for all supporting comments and help. It made me "think different", and that was what I needed. I now feel a bit like a fool not able to figure this out by myself, but a happy fool

Best Regards, and take care.
- Per Håkansson