Messages

Hello

I manage some OPNsense running as virtual machines on a couple of small Proxmox hardware at different remote locations. Whenever I need to update the OPNsense firmware remotely, I'm scared to lose the connectivity permanently because the update somehow might fail (even if it has worked fine every time so far, thanks).

Can I update the firmware on my virtual OPNsense from Proxmox? My thought is to run a Proxmox bash script to first shut down and back up the OPNsense, start it and send a firmware update command to the virtual OPNsense. After the update finish, maybe by letting the script wait for 15 minutes or more, the bash script checks connectivity 'through' the OPNsense to verify update success. If that connectivity check fails, the script restores the last backup of OPNsense (= revert all update) and restart it. I know all of this is possible except for the Proxmox command stuff, to send a (correct) update command into the virtual OPNsense machine.

The idea is to have the OPNsense updated, checked and if for some reason the update did fail, restore it to the previous version. Am I on the wrong track, can this maybe be done somehow within the OPNsense itself?

Sorry if I ask this question to the OPNsense forum, maybe it should go into the Proxmox forum instead. But I think I'm not the only one using Proxmox as the engine for OPNsense and also, I ask this forum to know what kind of command I need to send to have the update job done in every case. Like going from 20.7_4 to 21.1 without any prompts popping up. I need a command which will make the firmware update go through without asking interactive questions, is that even possible?

The goal is to have an automated update script running in the middle of the night that either will update the OPNsense or if there is any problem, revert to the original version. It should be a *stable* process which 'never' end up in a broken state. Either the update has succeeded, or it has not, but it should *always* be up in the morning. That's my goal.

Thanks for any help and any good though on this topic.

Best Regards
- Per Håkansson

Yes, yes, yes . . .  this seems to work great!! Thanks, mimugmail!

In the /usr/local/etc/rc.syshook.d/start directory, I copy the 90-cron.. job to 93-all-service-restart and edit the file to:

   #!/bin/sh

   echo -n "Restart all services: "
   configctl service reload all

This result in a startup output as follows:

   >>> Invoking start script 'all-service-restart'
   Restart all services: OK

. . . and guess what, as soon as the "Restart all services" start to execute, even before the "OK", my VXLAN starts to work :) This will fix my VXLAN remote bridge 'bug' for now.

Once again, thanks a lot. This is great. I now can go on to test encrypting my site to site VXLAN traffic.

Best regards
- Per Håkansson

Yes, yes, yes . . .  it seems to work great!! Thanks, mimugmail!

In the /usr/local/etc/rc.syshook.d/start directory, I copy the 90-cron.. job to 93-all-service-restart and edit the file to:

   #!/bin/sh

   echo -n "Restart all services: "
   configctl service reload all

This result in a startup output as follows:

   >>> Invoking start script 'all-service-restart'
   Restart all services: OK

. . . and guess what, as soon as the "Restart all services" start to execute, even before the "OK", my VXLAN starts to work :) This will fix my VXLAN remote bridge 'bug' for now.

Will this file survive an OPNsense update? . . or will the update/upgrade process wipe my new start file (93-all-service-restart)? Should I be worried to lose this fix after I update/upgrade OPNsense to next release? Does anyone know?

Once again, thanks a lot. This is great. I now can go on to test encrypting my site to site VXLAN traffic.

Best regards
- Per Håkansson

I want to execute a FreeBSD shell command every time OPNsense has started. I have tried to add a @reboot crontab job, but the crontab seems to be 'cleaned' after each reboot. My @reboot is no longer in the crontab -l after a reboot.

Maybe I can use RC in some way. If so, is there any good tutorial or easy example RC-file how to create an RC job in an OPNsense FreeBSD machine (I'm have not much experience using FreeBSD or Linux)? I did read that a bad RC job might halt the startup sequence. That made me a bit cautious playing around in the RC directories.

Maybe there is a plugin I can use from within OPNsense doing a startup script/shell command?

Best Regards
- Per Håkansson

No, I don't encrypt between A & B because both 'sites' are currently inside a single Proxmox. Right now, I only have this as a test environment. It's a single Proxmox, using 4*linux host as end-devices and 6*OPNsense to simulate a real site-to-site environment with VXLAN, Internet and everything. I use it to test out configuration to handle reboots and other problems, like Internet failure and NAT between site A & B. I try to get everything working. The goal is to solve everything before it is put in real production at real sites. But, yes, when I put the VXLAN over the actual Internet, I will for sure have to use encryption between A & B. I will test that to as soon I have got a stable VXLAN configuration.

The "Lock Prevention" doesn't make any difference. The stretched VXLAN is still not working after a reboot.

If I have 'bounced' the VXLAN interface and everything is working, it is no problem if I 'break' the Internet between OPNsense site A & site B. When the Internet starts to work again, my VXLAN also works. The only time I lose connectivity is when I reboot either of the two VXLAN OPNsense.

I found out a command I can run using the shell which solves the problem. It's "configctl service reload all". It might be a bit overkill, but it works. I did try to add the command to crontab with the @reboot. But after a reboot, somehow the line was removed from crontab.

Are there any better way to have OPNsense run a command at startup? I have tried to find a plugin to create a startup script somehow, I have tried to figure out how to use rc.d or rc.local, but I'm too bad in FreeBSD (or Linux) to figure it out. The best would, of course, be if the VXLAN could come up correctly, without me doing some 'ugly' fix running "service reload all". But having the OPNsense doing a "service reload all" just after OPNsense has started, works for me. It will only happen once every reboot, and that will not happen often . . . and the VXLAN function has already been down while OPNsense rebooted, so it does not matter if it's dead a couple of more seconds by the "service reload all" script.

Well, I will keep on trying to find a way to run the "configctl service reload all" at startup. I have tested so many OPNsense settings doing this the 'correct way', but everything I tried failed. So, my last hope is to do an 'ugly fix' :) by crontab, script or RC. Then, when the VXLAN survive a reboot, I will test out an IPsec/OpenVPN solution for the VXLAN traffic, even if the site is behind NAT. That should work, in theory. We will see which feature I 'hit' :)

All this is to replace our old solution currently in production. The OPNsense must be as stable (or better) then the current setup. It should also handle more and better encryption and handle more variation than the current solution. But I do like OPNsense, and so far, except for the VXLAN-'bug', it has worked great. I have hope.

Best Regards
- Per Håkansson

I have an interface on Site A and another interface on Site B, which I want to have layer 2 connectivity in-between. I add a VXLAN between Site A and Site B. In each OPNsense I bridge the interface with the VXLAN, this way, I have stretched the L2 network between the Site A interface and Site B interface. It works fine when I have it up and running. But each time I reboot the Site A or Site B OPNsense, I have to open VXLAN or the Bridge interface, manually press 'save - > apply' (no change), to have the L2 LAN working between Site A and B interfaces.

If I add another interface on Site A to the bridge, the two interfaces on Site A have no problem to send L2 packets after I restart the OPNsense at Site A. I have no problem using ping between Site A and Site B VXLAN source and dest. IP's after a reboot.

I guess that the bridge starts before the VXLAN has created its remote connection or something like that. This 'mess up' the L2 mac forwarding table in the bridge (or the VXLAN). This problem repeats the same way every time I reboot any of the two OPNsense. Every time I save the VXLAN interface (with no change), everything starts to work right away.

I have tried to cron a "Periodic interface reset" with "vxlan0" as the parameter on each Sites OPNsense, no luck. I also ssh to the box and manually did "ifconfig vxlan0 down" and then "ifconfig vxlan0 up", but it did not work either.

I would be grateful if anyone has a workaround (a corn configuration or maybe a bash script which can be executed at startup) or a suggestion which OPNsense setting I can try to change. I have done some test with different advanced interface setting (on the VXLAN, bridge and interface), but no luck so far.

I'm running the latest OPNsense.

Many thanks for any suggestion.

Regards
- Per Håkansson

Hi, Fabian

You were right this time too. It was a rule denying the active data channel creation, but not in my DMZ OPNsense. This time, the data channel was blocked on the client-side FW. Thats why I didn't see any block lines in the DMZ FW log.

But, I will not open the client-side OPNsense to all incoming high TCP port just to let active FTP works. I rather make active FTP fail. I know that the FW still needs a NAT/ALG/PROXY function to 'convert' the outside request into an inside connection. But somehow, this "allow-all-high-TCP-ports" open ups the FW for anyone who wants to try . . . or am I foolish and have misunderstood the whole concept of an FW? I usually don't even answer ICMP pings or sending any other ICMP message on my WAN.

My conclusion: I have an FTP server-side proxy function allowing both passive and active FTP sessions, that's good. I will not open up the WAN side of my client OPNsense for high order TCP ports to support active FTP.

Is there any discussion having FTP proxy create temporarily rule's for the data channel? The proxy software has all the information it needs looking at the control channel to make such a temp rule in the firewall rule-set. That would be a very 'smooth' feature . . . if you ask me (even if this topic tells you I'm not the best to figure stuff out :)

Once again, Fabian, thanks a lot for all supporting comments and help. It made me "think different", and that was what I needed. I now feel a bit like a fool not able to figure this out by myself, but a happy fool :)

Best Regards, and take care.
- Per Håkansson

Thanks, Fabian

Made me thinking. Rules for data channel too?? I believed that was the job for FTP proxy, to temporarily create data channel rules from what's in the control channel. But after your comment, I did realise I maybe had 'hoped' too much of the FTP proxy.

So, after once again clean out my OPNsense from everything regarding FTP proxy, I reinstall the FTP proxy again, add NAT to 127.0.0.1 and add a port 21 rule on the WAN as stated in the documentation.

This time, I also added an "any any" rule on my FTP DMZ interface letting the FTP server do whatever it likes (I don't like that, but what the heck, it's a test). I also add a rule on the WAN interface allowing specific vsftpd passive TCP destination ports to pass to the FTP server. This WAN rule was something new to me (and I'm not too fond of it from a security point, even if there are no inbound NAT so that would be pretty safe anyway I think).

And now passive FTP works, yeah great!!! I now have understood that the FTP proxy adds a temporarily NAT for the data channel to the FTP server. It does *not* add a firewall rule on the other hand. This 'auto-NAT but not auto-rule' function is useful to know.

When I try to use active FTP, my firewall log show that the FTP data channel is allowed from my FTP server out to the external FTP client, inside-ftp-serv:20 -> extIP.23456 due to my "any any" rule. But it still not working. I will have to do the same packet capture for this problem. I suspect that the data channel port number is not translated correctly from inside to outside (but its a guess and I have to verify it). I will add whatever I find out to this post when I have captured the traffic.

Once again Fabian, thanks for 'open my eye' on how the FTP proxy works. I am a network guy mostly working with Cisco ASA, and I build my assumption from it's FTP inspection. Sorry for not 'thinking it through'. I hope my problem with active FTP also depends on 'wrong thinking' :)

Best Regards
- Per Håkansson

[INTERNET FTP DMZExtClient]

This traffic 'drawing' from both side of the OPNsense/FTP proxy plugin, indicate that it fails to 'prepare' for the FTP data channel (if you ask me). It handles the FTP control channel as expected.

        INTERNET                                FTP DMZ
ExtClient(InetIP)-(DHCP-DynDNS)OPNsense(10.1.1.1)-(10.1.1.2)FTP Srv(vsftpd)
                                 with
                               FTPproxy
                                plugin

  TCP port 21 setup and FTP ->!        ! TCP port 21 setup and FTP ->
  login works fine every time !        ! login works fine every time

                          Then this happens

         FEATreq.dstport21--> !        !
                              !        ! FEATreg.1->.2.dstport21--->
                              !        !<--EPRT,EPSV,MDTM,PASV,
                              !        !REST S.,SIZE,TVFS,UTF8
        <-EPRT,EPSV,MDTM,PASV,!        !
        REST S.,SIZE,TVFS,UTF8!        !
        EPRT-1-InetIP-55838-->!        !
                              !        ! EPRT-1-.1-54376--->
                              !        !<--EPRT-Succ.Consid.EPSV
      <--EPRT-Succ.Consid.EPSV!        !
                       LIST-->!        !
                              !        ! LIST-->
                              !        !
            -------           !    -   !<-SYN.1.dst54376.2.src20
                              !    !   !<-SYN.1.dst54376.2.src20 (Retransm. 1s)
            Nothing           !    !   !<-SYN.1.dst54376.2.src20 (Retransm. 2s)
            happens           !   30s  !<-SYN.1.dst54376.2.src20 (Retransm. 4s)
             here!            !    !   !<-SYN.1.dst54376.2.src20 (Retransm. 8s)
                              !    !   !<-SYN.1.dst54376.2.src20 (Retransm. 16s)
            -------           !    -   ! ICMP.1 host .1 unreach. -> ?From .1?
                              !        !
                              !        !<--Src.2.21 FTP 425 Failed conn
                              !        ! TCP ack from .1 --> (?Why ICMP unreach?)
     <--Src.21 425 Failed conn!        !


It does not matter if I do it with passive or active FTP. The data session does not come through. Still, the control channel works both with passive and active. As you can see from my 'drawing', the FTPproxy does its job for the control channel. It does change the IP src/dst in both directions, and even change the IP inside the EPRT FTP command as it should. It's all about the FTP proxy fail to prepare to open the 'data channel' port in the firewall together with a 'temporary NAT' (= my guess).

At least, this is how I guess an FTP proxy should work. Maybe I have missed something, but I have tried all FTP proxy settings I can come up with, like source IP address, and rewrite source port. I have moved the 'FTP rules/NATs' to the top in each list, I have created a 'dummy' rule to allow all traffic out from the FTP server. It still can't handle the dynamic data channel TCP port session.

I also run another FTP proxy on another OPNsense. Its a 'client-based' configuration, no reverse address/port, which works fine when I connect to public FTP servers. My problem starts when I want to have an FTP server (vsftpd) on the inside (FTP DMZ) of an OPNsense setup. I have tried to remove and reinstall the FTP proxy plugin. I have rebooted before, between and after all my tests. Still the same result.

I have tcpdump in the FTP server, packet capture in OPNsense and Wireshark on the outside to understand where the FTP fails. Maybe there are some log's I have not found, which might tell me something more useful. But as a non-Linux guy, I have so far missed such log's :(

One more thing, the OPNsense have ten inside interfaces (opt's), one for each DMZ servers it handles. It also has an internal interconnect link (lan) for management . . . and of course the WAN outside Inet interface (wan). All interfaces are 'virtual' created as Proxmox VLAN's (= OPNsense does not know each interface is a VLAN tag in the Proxmox guest setup). From the OPNsense perspective, it has 12 real interfaces (vtnet 0-11). I can't see why this should have anything to do with the FTP data channel, but a computer is a computer, a sensible machine with its own will :) So better to tell you the setup right away. I have tested this setup from two different FTP client software's, from an iPad and a PC. Both fail to connect the FTP data channel, even if the control channel works fine.

FTP proxy is set up as mention in the OPNsense documentation. A 127.0.0.1/8021 proxy with a NAT 'catch' on the WAN interface with a FW rule allow port 21 in on the WAN. And as you can see, it seems to work fine as long as it only handles FTP control traffic (port 21).

So, any suggestions? Should my multi-interface FTP proxy setup work or should I go in some other direction? What should be my next step? Maybe I can use NAT only in some way, without the FTP proxy plugin, by allowing a narrow TCP port range to be NAT'ed to the vsftpd DMZ server. Any suggestions would kindly be accepted. I cant come up with more things to investigate, and I can't figure it out. If I can't get any good advice from this query, I have to start looking somewhere else then OPNsense . . . and I don't want to leave OPNsense. It has so much to offer and drop it because of such a 'small' thing as an FTP problem hurts.

Best Regards and thanks for all help I can get.
- Per Håkansson

Hello all

I'm new to OPNsense (= maybe I, therefore, ask a stupid Q). I like what I have seen of the OPNsense so far. But I have run into a 'problem', which I hope have an easy fix. I have been trying some settings, but all have failed. I'm running the latest version of OPNsense.

This is a simplified setup description:
NET-A 192.168.0.x/24 <-> LAN [OPNsense] WAN <-> NET-A should look like 10.11.12.x/24 on this side of the OPNsense for all other networks (routed).

The best setting I can come up with is to use 1-to-1 with the following parameters, but it does not work:
WAN
BINAT
Ext-net 10.11.12.0/24
Source lan/24
(BTW: I use manually on my outbound NAT if it makes any difference)

My question:
Do I have to add rules on WAN and LAN interface allowing all possible traffic which will go through this 1-to-1 NAT (probably)?
Should I go for some of the other two NAT options (port forward or outgoing) (probably not)?

Something more, so I understand OPNsense a bit more:
In which 'order' will a packet 'traverse' all functions in OPNsense? Is it, interface in-filter, forwarding, NAT port forwarding, NAT 1-to-1, outgoing NAT, VPN, Interface out-filter? Where do all the 'inspect' plugin add-in to this step-by-step functions? I assume that the NAT rules are served top to bottom, and exit if it 'hit' a rule . . . but will it exit all NAT or just the NAT currently in process? Like if it hit a port forward NAT, will it exit all NAT checking or will it jump to do 1-to-1 NAT checking?

Sorry for all (stupid) questions, but I like to understand this software and how it works (below) the nice GUI. It's needed to do proper troubleshooting I think.

Best Regards
- Per Håkansson