Messages

Hiya,

Our set-up is similar to https://forum.opnsense.org/index.php?topic=18732.msg85748#msg85748 except we have a single switch.

One of two problems is that the FTP proxy sometimes connects to the physical address of the firewall instead of the virtual address.

On a Windows laptop, connect to the internet and not connect to the intranet, I can create a file on the FTP server like this:

Code Select Expand

C:\>echo 123 | curl -T - --ftp-create-dirs --ftp-pasv -u ftpuser001:Password ftp://ftp.example.com/test-dir/test-file.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     6    0     0    0     6      0     29 --:--:-- --:--:-- --:--:--    29

And can list the file with:

Code Select Expand

C:\>curl --ftp-pasv -u ftpuser001:Password ftp://ftp.example.com/test-dir/
-rw-r--r--   1 ftpuser001 ftpusers        6 May 23 06:44 test-file.txt

Something the listing works and sometimes it times out

When it times out, I see this in the ftp server log

Code Select Expand

2022-05-23 10:01:27,224 s2 proftpd[6317] s2.example.net (firewall.example.net[10.99.0.1]): SECURITY VIOLATION: Passive connection from foreign IP address 10.99.0.2 rejected (does not match client IP address 10.99.0.1).

The master firewall has the physical IP 10.99.0.2 and the virtual IP 10.99.0.1.
The FTP server (s2) has the physical address 10.99.0.102 plus some aliases. The aliases are for KDC, LDAP and a log host.

I've included the IP aliases for completeness, but don't think they are affect the source IP.

Code Select Expand

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.99.0.102  netmask 255.255.255.0  broadcast 10.99.0.255
        ether ac:1f:6b:76:e7:2e  txqueuelen 1000  (Ethernet)
        RX packets 402772438  bytes 125251420679 (116.6 GiB)
        RX errors 0  dropped 0  overruns 779331  frame 0
        TX packets 384462758  bytes 191478460768 (178.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xdf200000-df27ffff 

eth0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.99.0.31  netmask 255.255.255.0  broadcast 10.99.0.255
        ether ac:1f:6b:76:e7:2e  txqueuelen 1000  (Ethernet)
        device memory 0xdf200000-df27ffff 

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.99.0.34  netmask 255.255.255.0  broadcast 10.99.0.255
        ether ac:1f:6b:76:e7:2e  txqueuelen 1000  (Ethernet)
        device memory 0xdf200000-df27ffff 

eth0:2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.99.0.250  netmask 255.255.255.0  broadcast 10.99.0.255
        ether ac:1f:6b:76:e7:2e  txqueuelen 1000  (Ethernet)
        device memory 0xdf200000-df27ffff 

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 18713387  bytes 8392738213 (7.8 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 18713387  bytes 8392738213 (7.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 10.8.0.1  netmask 255.255.255.255  destination 10.8.0.2
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 100  (UNSPEC)
        RX packets 78772655  bytes 43780025295 (40.7 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 49179619  bytes 6922383213 (6.4 GiB)
        TX errors 0  dropped 27101 overruns 0  carrier 0  collisions 0

Hopefully someone knows how to solve this, TIA
-spider

[1. What is the purpose of the PF sync ? Is it only for not breaking the user sessions when the failover starts ?]

Hi,

I'm no expert, but may be this helps a little.

1. What is the purpose of the PF sync ? Is it only for not breaking the user sessions when the failover starts ?
The pfsync interfaces perform two tasks, to pass the states of the connection to the backup firewall and the transfer the XMLRPC configuration.

2. About the interfaces, as far as I understand this, I'll have to create them by hand on each boxes, right ?
Yes

3. If yes to the previous question, how does it work with VPN interfaces ? Would I have to also make the assignment by hand ?
Don't know

4. About the replication, as I understand it, it's only from the main box to the slave box. Does it mean I can have specific rules and configurations on the slave box, and if they are, for example on dedicated interfaces (which don't exist on the main box), they will not be overwritten by the replication ?
You can choose what is replicated down to a specific firewall rule

5. How does work the setting replications with plugins ? For example, I have a telegraf supervision, FRR configurations, reverse proxy, few VPN (OpenVPN, Wireguard and Ikev2), etc.
WireGuard works fine is there are no keep alive packets on the server side. Make little sense to have keep alive packets on an endpoint without an endpoint address. OpenVPN works just fine too.

If your backup firewall is the APU then there is something special you'll need to do, See the how-tos. https://docs.opnsense.org/manual/hacarp.html and https://docs.opnsense.org/manual/how-tos/carp.html

HTH
-spider

VIPs are needed where you want a continuity of services when the primary is offline.

For the businesses I work with, we want continuity of all services during reboots, hardware failures, etc.  So every interface we have also has a VIP with it.

As someone who has worked at businesses where VIPs aren't "always" used for all situations, it *really* sucks when you have a network problem, and you're troubleshooting and the lack of VIPs sometimes limit your options for troubleshooting.  It also adds a lot of confusion because you're constantly trying to figure out what is broken by design (aka there isn't a VIP) and what is broken by fault.

Many thanks for the tip, it makes a lot of sense. It's not always obvious when the backup is running as the master.

What was failing was an OpenVPN pass-through (OpenVPN server on another machine) where the connection was dropping out and coming back every couple of minutes, I guess that the packets were being routed through the master and backup firewalls.

Thanks again
-spider

No, in my setup for reasons of simplification, only LAN, WAN and Guest have a virtual IP. OpenVPN doesn't. Of course there will be not HA for OpenVPN, but I don't care.

Thank you, that's good to know. I also don't care if all the services are available when the master is off-line.

I'll try giving the openvpn client a VIP to see if this allows this VPN to work reliably.

-spider

Hi all,

Do all interfaces need a Virtual IP address?

I've just completed installing a high availability configuration and a OpenVPN client seems to be on-line for a couple of minutes, off-line for a couple of minutes and back on-line again.

The OpenVPN server is on a Linux host and not on the OPNsense host, so the traffic is passing through the OPNsense host. The OPNsense host has a client connection to the OpenVPN host to allow traffic to LAN hosts to access the VPN hosts and has an interface for this OpenVPN client.

There is also a guest subnet for Wi-Fi clients that is connected to both in the HA cluster, which, I think, would need a VIP. Currently on the LAN and WAN interfaces have VIPs

Hope this makes sense and thanks,
-spider

Some network history was causing this to fail. Originally, we have an ISP from the phone company, and then we replaced this ISP with another ISP.

What I did was to create another WAN interface so that I could activate the new internet connection with minimal downtime. After the change over, I kept the original WAN settings in case we needed to fall back to the original ISP. This weekend, I implemented high availability and noticed that a test WireGuard host was getting a handshake. Then the penny dropped, and I realized that the same IP address was on both firewalls.

Removing the static IP from the main firewall (master and backup) for the old WAN connection, then the second firewall worked correctly.

Code Select Expand


Firewall 1 (Master+Backup)
WAN - static IP of ISP1 but no cable
WAN2 - static IP connected to the ISP2
Firewall 2
WAN - static IP connected to ISP1


The fix was to change the WAN interface from a static IP to None.

-spider

I figured this out for myself, purely by chance. There was some history on the other side of the WireGuard connection that was using the same IP address.

Hi

I have these rules


I don't think you need to create an interface for an OpenVPN server.

There is more documentation at https://docs.opnsense.org/manual/vpnet.html#openvpn-ssl-vpn

cheers
-spider

Hi,

I wonder if your problem is similar to mine, it sounds like it.

Code Select Expand

                                                                               ********   
    +-------------------+                      +-------------------+          **      **
    | OPNsense Test     |  WAN ---------> LAN  | OPNsense Firewall | WAN --> * Internet *
    |                   |  LAN            DHCP |                   | static   **      **
    +-------------------+   | 10.10.0.1        +-------------------+           ********   
                            |
                            v 10.10.0.100
                +-------------------+
                | Workstation       |
                |                   |
                +-------------------+

In the above diagram, the OPNsense Test box is set up as a WireGuard client. As documented in https://docs.opnsense.org/manual/how-tos/wireguard-client.html

If the OPNsense Firewall is replaced with an Asus 4G router, then WireGuard connects correctly. I'm wondering if anything needs to be configured in the OPNsense Firewall box, As the OPNsense Test box initiates the connection I don't think it needs port forwarding runs.

Any help would be gratefully applauded.
-spider

For dhcpd you need to configure its builtin synchronisation mechanism. On the master node in e.g. Services > DHCPv4 > LAN put the IP address of the backup node into the "Failover peer IP" field.

Thank you for this information. What I did was to download the example files and edited the interfaces and IP addresses. Then restored these to two pc-engines boxes.

Second, for configuration backup I found the os-git-backup plugin to be a really good way to get a versioned configuration history. If you have multiple administrators, each with their own login, it will even log who made the change.

Nice tip, thanks,

Never sure how to use git for what I need, for example I wanted to find the last configuration file before moving from a pc-engines box to a Supermicro box and I knew there was a new interface on the new box. I could use grep and retrieve the previous file. With git I wouldn't know where to start. Personally, I find IPv6 addresses, git commit version numbers and other GUID type numbers difficult to read and consequently try not to use them.

Cheers,
-spider

Hi all,

I reckon this must be blocked by the ISP, I need to figure out how to tell if the IPS is blocking UDP ports and if so, which ones.

The latest version of UDT file transfer is from https://github.com/Haivision/srt.git

I set up a port forwarding rule to pass UDP traffic on port 4201 to my workstation (all our company workstations run Debian) which then is listening for connections using the command: srt-file-transmit -v srt://4201 file://

Then on a workstation behind an OPNsense firewall connected to a 4G router I ran the command: srt-file-transmit -v file:///usr/src/srt/README.md srt://example.com:4201

What I see on the sending machine is:

Code Select Expand

SOURCE type=file, TARGET type=srt
Extract path '/usr/src/srt/README.md': directory=/usr/src/srt filename=README.md
SRT parameters specified:
        transtype = 'file'
Opening SRT target caller on example.com:4201
Connecting to example.com:4201
Target connected (caller)
Upload: 1456 --> 1456
Upload: 1456 --> 1456
Upload: 1456 --> 1456
Upload: 1456 --> 1456
Upload: 377 --> 377
File sent
Sending buffer still: bytes=6201 blocks=5
Buffers flushed
SrtCommon: DESTROYING CONNECTION, closing sockets (rt%756710371 ls%-1)...
SrtCommon: ... done.

on the receiving machine I see the following:

Code Select Expand

SOURCE type=srt, TARGET type=file
Extract path '': directory=/data/intern/. filename=
SRT parameters specified:
        transtype = 'file'
Opening SRT source listener on :4201
Binding a server on :4201 ...
listen...
Event with status 3
accept...
connected.
Source connected (listener), id [README.md]
Event with status 5
Writing output to [/data/intern/./README.md]
Download: --> 1456
Event with status 5
Download: --> 1456
Event with status 5
Download: --> 1456
Event with status 5
Download: --> 1456
Event with status 5
Download: --> 377
Event with status 6
Connection closed, reading buffer remains
Download COMPLETE.
SrtCommon: DESTROYING CONNECTION, closing sockets (rt%1011421236 ls%-1)...
SrtCommon: ... done.

So, everything is working just file.

As we have an old internet connection, that was used for both internet access and telephony I thought it would be a good idea to use this as a listener to receive incoming files (typically 200 GB) or video streams. So, my first test was to use this connection to send the file to my workstation, this time I see:

Code Select Expand

SOURCE type=file, TARGET type=srt
Extract path '/usr/src/srt/README.md': directory=/usr/src/srt filename=README.md
SRT parameters specified:
        transtype = 'file'
Opening SRT target caller on example.com:4201
Connecting to example.com:4201
Target disconnected
SrtCommon: DESTROYING CONNECTION, closing sockets (rt%643233832 ls%-1)...
SrtCommon: ... done.
So, it is refusing to connect to the same machine listening to the same port.

Does anybody have an idea what would cause this to fail?

Thanks

https://docs.opnsense.org/manual/hacarp.html#automatic-replication

https://docs.opnsense.org/manual/hacarp.html#status

Many thanks for this information, helpful, thanks.

In the config.xml the DHCP reservations are under the <dhcpd> so they do get replicated to the backup server when the DHCPD box in the High Availability setting is checked. I guess that I hadn't clicked the Upload icon on the High Availability status page.

As a slight aside

I use cron from a server to back up the configuration files regularly, something like this:

Code Select Expand

15 6,12 * * * root scp root@office:/conf/config.xml /data/storage1/backup/opnsense/config-office-$(date "+\%Y\%m\%d-\%H\%M").xml > /dev/null 2>&1

Then another cron task removes the duplicated back ups.

Code Select Expand

#! /bin/bash
cd $(dirname "$0")
PATS='config-office-*.xml config-opcase1-*.xml config-opcase2-*.xml'
LAST=""
for i in ${PATS}; do
    #echo $i;
    if [ "$LAST" == "" ]; then
        LAST="$i"
    else
        SUMS=($(sha256sum $LAST $i 2> /dev/null))
        #echo SUMS: "${#SUMS[@]}" "${SUMS[0]}" "${SUMS[2]}"
        if [ "${SUMS}" == "" ]; then continue; fi
        if [ "${SUMS[0]}" != "" -a "${SUMS[0]}" == "${SUMS[2]}" ]; then
            echo keep ${SUMS[1]} remove ${SUMS[3]}
            rm -v ${SUMS[3]}
        fi
        LAST=$i
    fi
done


You need to click small cloud button at this status page

Thanks, after two days I figured it out, meanwhile it was rather frustrating. The wiki page is excellent but didn't say much about synchronization.

Still haven't figured out if it is possible to synchronize DHCP reservations. If it is not possible, then I don't need to spend time on this part.

We use DHCP reservations for almost all the hosts, even hosts with static IPs.

Many thanks
-spider

Hi,

Can you access the router from your firewall?

For example:

Code Select Expand

# curl 10.99.128.1
<HTML><HEAD><script>top.location.href='/Main_Login.asp';</script>
</HEAD></HTML>

If you can, then maybe you need a static route.

Sorry, if this is a daft reply, I'm still a bit of a n00b when it comes to networking. Sometimes just knowing if works is enough for me.
-spider

Hi,

Is it possible to sync DHCPv4 IP reservations?

The other part of the question, about seeing if the XMLPRC sync has worked, has been resolved. The system and audit logs shows the status of the sync.

The reason why this was not visible when I asked is because the sync had not been executed. This is partly due to the interface being less intuitive than it could be.

shows the first row of the status page's service and the upload button on the right-hand side need to be clicked. This is not as clear as a button used in the Virtual IP status page.


Thanks,
-spider