Messages

AdGuard Home is available on OPNsense and does that quite nicely, too.

Indeed it does, I hadn't looked very closely at it before. However I would still need a second, independent instance to ensure continuity of service while fiddling. 

I have 44 lists which gives me about 600 000 blocked domains.

I have only 4 lists in unbound blacklist category and they gave me about 900k blocked donains. Everything that can be blocked via domains can be blocked as it is, you don't need an additional service for that.

OK, will you please come and teach my wife how to unblock a domain in unbound?
Seriously though, the Pi-Hole interface if far more user-friendly, you can easily create groups to follow different rules, whitelist (or blacklist) specific domains from the activity log etc.

Changing from the on-board NIC (Fujitsu S920) to another network port in the router seems to have resolved the issue.  The router and switch indeed couldn't auto-negotiate.  Perhaps a driver issue?

No help for you, but here's a similar weird one..
My Asustor NAS kept on dropping the connection (2.5 Gb/s) to the switch (cheap Chinese one). Rebooting either the switch of the NAS would bring it back up. It seemed to fail when I was hammering the NAS (backing up to the cloud at 2.5 Gb/s) so I suspected that something was overheating somewhere. I ordered a new switch. In the meantime one of the disks in the NAS failed with the "click of death", so I swapped it out. No NIC problems ever since so I have a spare switch!
Maybe the failing disk was drawing more current or sending spurious noise back over the SATA or power lines, who knows?

Pi-Hole will do most of what you want.
I run two (belt and braces) on a couple NanoPi NEO single board computers, they have a wired ethernet port and cost about 13€ (plus old spare phone chargers to run them). There is a lovely operating system called diet-pi which is light and comes with pre-configured modules which you can install, pi-hole is one of them.
If you're into 3D printing I have designed a case for them.
There are many blocklists available for pi-hole, the software deals with duplicates for you. I have 44 lists which gives me about 600 000 blocked domains.
The interface has a "turn off for X minutes" option for when the Missus wants to go to that dodgy shopping site.
I have also built a "Pie-Stop" button (from Planet Kris) which uses an ESP-01S to send the "turn-off" message via WiFi (pi-hole has an API available).
If you are tempted to run pi-hole in a VM somewhere it will work perfectly, but remember that if you turn off the VM host for some reason you will lose DNS for your whole network.
Hope this helps.

When both of those protocols are slow upstream, it seems like the BDP is high on your connection. You could use traffic shaping or try disabling "Optimize connection buffer size" there. This is explained here.

Thanks for these pointers. Changing the encryption algo was the only one that made a difference though, doubled upload throughput by moving to Blowfish. This could mean that there is a problem on the client side. I will look into traffic shaping, but I'm not hopeful.

UPDATE
Using OpenSSH on the server and WinSCP on the client SFTP gives 48MB/s download (about 50% of FTPS) and 6-10MB/s upload (an improvement, but far from what it should be).

Thanks for the reply.
I'll try SFTP, but the servers are used by friends for whom anything more complex than a user name and password is a bit frightening. However if SFTP solves the problem I'll drag them into the 21st century.

Hi,
Bit of a noob here so please be gentle.

OPNsense 24.7.10_2-amd64
FreeBSD 14.1-RELEASE-p6
OpenSSL 3.0.15

Running on an Intel N100 mini PC with 4x i266-V, 8GB DDR4, 250GB NVMe Gen 3x4.
igc0 connected to a 2.5Gb/s fibre modem/router in bridge mode (French Freebox Ultra)
igc1-3 bridged for LAN

I am connecting to ftp servers on a couple of remote dedicated servers (Scaleway), both running Windows Server in an ESXi VM. One server is Win 2012 IIS, the other Filezilla under Win 2019. Both servers have 1000Mb/s connections.
Both servers have a range of 1000 open ports defined and opened in Windows FW.

FTP and FTPS downloads from either server are rapid, generally rising to 100 MB/s.
Uploads to the servers are dire, maxing out at 5 MB/s, however I can do  several concurrent uploads to a combined limit of about 25MB/s.

Behaviour is the same whatever the client software (SmartFTP, Filezilla, lftp).
iperf3 shows symmetrical normal speeds between servers and home PC
Ookla shows symmetrical normal speeds on the client and servers.
Debian cli ftp shows the same poor upload speeds.

Here is a typical example, connection from local Debian to distant Filezilla on Windows 2019:

Code Select Expand

lftp Me@www.xxx.yyy.zzz:/> put "The Flint Street Nativity (1999)-1.mp4"
723355399 bytes transferred in 152 seconds (4.54 MiB/s)
lftp Me@www.xxx.yyy.zzz:/> !rm "The Flint Street Nativity (1999)-1.mp4"
lftp Me@www.xxx.yyy.zzz:/> get "The Flint Street Nativity (1999)-1.mp4"
723355399 bytes transferred in 6 seconds (108.66 MiB/s)

Here are the corresponding Filezilla server log entries, nothing exceptional as far as I can see:
UPLOAD

Code Select Expand

<Date/Time> Info [Type] Message
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Command] FEAT
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Response] 211-Features:
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Response] 211 End
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Command] AUTH TLS
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Response] 234 Using authentication type TLS.
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Command] OPTS UTF8 ON
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Response] 202 UTF8 mode is always enabled. No need to send this command
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Command] USER Me
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Response] 331 Please, specify the password.
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Command] PASS ****
<11/12/2024 16:49:13> FTP Session 41 <IP Addr hidden> [Response] 230 Login successful.
<11/12/2024 16:49:13> FTP Session 41 <IP Addr hidden> Me [Command] PBSZ 0
<11/12/2024 16:49:13> FTP Session 41 <IP Addr hidden> Me [Response] 200 PBSZ=0
<11/12/2024 16:49:13> FTP Session 41 <IP Addr hidden> Me [Command] PROT P
<11/12/2024 16:49:13> FTP Session 41 <IP Addr hidden> Me [Response] 200 Protection level set to P
<11/12/2024 16:49:17> FTP Session 41 <IP Addr hidden> Me [Command] TYPE I
<11/12/2024 16:49:17> FTP Session 41 <IP Addr hidden> Me [Response] 200 Type set to I
<11/12/2024 16:49:17> FTP Session 41 <IP Addr hidden> Me [Command] PASV
<11/12/2024 16:49:17> FTP Session 41 <IP Addr hidden> Me [Response] 227 Entering Passive Mode (nnn,nnn,nnn,nnn,30,216)
<11/12/2024 16:49:17> FTP Session 41 <IP Addr hidden> Me [Command] STOR The Flint Street Nativity (1999)-1.mp4
<11/12/2024 16:49:17> FTP Session 41 <IP Addr hidden> Me [Response] 150 Starting data transfer.
<11/12/2024 16:51:22> FTP Session 41 <IP Addr hidden> Me [Response] 226 Operation successful
<11/12/2024 16:51:22> FTP Session 41 <IP Addr hidden> Me [Command] MFMT 20241209201305 The Flint Street Nativity (1999)-1.mp4
<11/12/2024 16:51:22> FTP Session 41 <IP Addr hidden> Me [Response] 213 modify=20241209201305.000; /The Flint Street Nativity (1999)-1.mp4


DOWNLOAD

Code Select Expand

<Date/Time> Info [Type] Message
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Command] FEAT
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Response] 211-Features:
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Response] 211 End
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Command] AUTH TLS
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Response] 234 Using authentication type TLS.
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Command] OPTS UTF8 ON
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Response] 202 UTF8 mode is always enabled. No need to send this command
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Command] USER Me
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Response] 331 Please, specify the password.
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Command] PASS ****
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Response] 230 Login successful.
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Command] PBSZ 0
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Response] 200 PBSZ=0
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Command] PROT P
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Response] 200 Protection level set to P
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Command] TYPE I
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Response] 200 Type set to I
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Command] SIZE The Flint Street Nativity (1999)-1.mp4
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Response] 213 723355399
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Command] MDTM The Flint Street Nativity (1999)-1.mp4
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Response] 213 20241209201305.000
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Command] PASV
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Response] 227 Entering Passive Mode (nnn,nnn,nnn,nnn,28,212)
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Command] RETR The Flint Street Nativity (1999)-1.mp4
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Response] 150 Starting data transfer.
<11/12/2024 16:42:45> FTP Session 40 <IP Addr hidden> Me [Response] 226 Operation successful


I am fairly sure (but not 100%) that the Windows servers or ESXi are not at fault since 'normal' speed tests seem to show good symmetric transfers.
Similarly FTP clients on Windows and Debian seem to show the same problem, but speed tests are good.
ntttcp test is also good.
The only elements left are OPNSense or my ISP's modem/router. I'm fairly sure that if the modem was  throttling FTPS uploads we would have heard of it by now.
Looking at OPNSense's cpu usage during transfers gives no clues, it's about 8% for upload and download.
I am not at all at ease with firewall rules and monitoring. I've reached my diagnostic limits here. Has anybody come across a similar problem or have an idea of what to look at next, please?
Many thanks in advance.