Messages

Or Asus P10S-I. A bit more money but you get more onboard LAN ports and more CPU power (and choice). Not quite so niche so not so difficult to purchase either.

Well I guess it's hard to know what's what in the whole chain of ISP, router and network including WiFi.. so maybe I'm reading too much into it. But with the combination of an ISP well-known for reliability and speed, gigabit fiber, opnsense on i5 and then WiFi on Ubiquiti access points it's more the *absense* of any sort of discernible lag or uneven performance, ever, unless it is evidently at the other end (e.g. Apple iCloud...). And that includes most of the family computers that accesses the web through Squid on opnsense so that I can filter content (using one of those public lists) to block some of the really bad stuff. Opnsense also running its own caching nameserver, dhcp and ntp.

I frequently find on other networks that performance is much more uneven or even categorically slower even if on paper it should be the same of faster.

But as mentioned, never easy to know what is what... but I like headroom. ;-)

I like the idea of it (Suricata), although must confess I don't think I've ever caught anything with it ;-) (lots of false positives though)

Just another point worth considering too.. to whatever extent you plan to run services on the box beyond vanilla routing and firewall, like caching & filtering web proxy (squid), dns, vpn (site-to-site, outbound or road warrior) etc - it's nice to have them sharp & snappy. I'm kind of used to it from back home and don't think about it but can often tell the difference when I'm on other networks.

For what it's worth, I'm running on an i5-5250U and Intel NICs and get about 600-700MBit with Suricata and some rules enabled. Easily saturate gigabit without Suricata.

I wouldn't have gone below i5 with gigabit wan. Next machine is going to be beefier to ensure some headroom, likely Xeon. Probably still possible to keep it quiet, with careful selection of chassis and coolers.

(sorry, regarding gigabit and ids, I must have mixed up the threads. ;-) anyway, take the advise for what it is - if you want more performance, look at those other options, and Qotom is likely to give you the best bang for the buck depending on where you live and whether you can order them direct from AliExpress or not... I'm using an i5 mode with great success for my gigabit wan.)

In that series, and with the performance requirements for gigabit and ids I would probably have looked at this model instead: https://www.thomas-krenn.com/en/products/application/opnsense-firewalls/les-network-plus-opnsense.html

Or alternatively a Qotom Q355G4, can be found cheaper

Or alternatively a custom build around for example a Asus P10S-I mini itx board. You'd have to add a dual or quad Ethernet card (PCI).
https://www.asus.com/Commercial-Servers-Workstations/P10S-I/

Yes. Qotom 355G4 with i5, opnsense and symmetric gigabit WAN. Without IDS/IPS I easily saturate full gigabit, with Suricata and my current rule set I get approx 800Mbit down and 700 up.

Although I have bridge set up as a unmanaged switch now, I am interested if there is a more efficient way to set this up.  Not sure if the unmanage switches are smart enough i  that they only route traffic onto the port where the device is connected...or it 'broadcasts' it on all ports hoping one has the device with that IP is connected.   May i  the future need to make work smarter to reduce network loading (especially if I add a few more IP cameras).

Small (5 port) gigabit hardware switches are very cheap these days and (at least Netgear and D-Link) very reliable too. Those definitely function as you mention ie keep tables of MAC addresses reachable at each port and route traffic accordingly. Old 'hubs' from the days of 10 Mbit Ethernet used to broadcast indiscriminately but that was quite a while ago.

Not sure if there are many strong cases for software bridging these days given how cheap the hardware is? At least if you're trying to minimise complexity and chances of things going wrong in connection with software updates etc...

I've got that Qotom model too (i5), bought from Ali Express. Similarly, gets a bit hot once the ambient approaches 30 C. However it was quite cheap and irrespective, I found fanless design with 4 Intel NICs pretty hard to come by other than as an exotic (expensive) custom build. From that persp I think the Qotoms are pretty unique.

Because of where the fiber connects to my home, and wiring constraints, I keep the router in the bedroom so the noise level is pretty important for me. If that was less of a consideration I would have used a cheap second hand business desktop instead with a bunch of NICs on PCI cards.

Or as a VM... I've got an ESXi server in the cellar and that is *not* quiet ;-)
That would need more cabling though, or exotic VLAN configuration

Rather than spending more money on trying to make this work, wouldn't it be wiser to replace the NUC with a quiet small form factor slim PC with dual Intel nics on a PCI-card? They can be very cheap second hand. Or one of the Qotom boxes. Your Realtek adapter may be the problem here (poor drivers in FreeBSD) and I don't know if you can find a well supported one (Intel) on USB in the 1st place. Also depending on what NUC you've got, but the more recent ones have pretty beefy graphics etc - better suited as a HTPC than a headless router...?

+1, got one of those (in i5) too and it works well. Runs a little hot in its cupboard in hot summer ambient temperatures (30ish C ambient) but solved that with small USB fan against the case. Sort of defeats the purpose of the otherwise fanless design, but oh well. Haven't tried replacing the CPU thermal paste.

Speed wise I can push approx 800Mbit with IDS enabled. Haven't speed tested VPN.

Gazd25, you may want to avoid storing passwords in scripts as well, not the least kept on hosts that are also public web servers. You could avoid passwords by using certificates, although that still doesn't change the fact that an attacker who gains control over the web server can still access the router. How about doing it the other way around, and pushing the certs from opnsense onto the windows server as opposed to pulling them? If only you can figure out a mechanism for that in Windows, I guess ssh may be tricky, maybe an ftp server? Scripting and automation is easy on the unix side.

Ok here they are.

fetch-push.sh (runs weekly on a command & control Linux VM on the lan via cron):

Code Select Expand


#!/bin/bash
function age() {
   local filename=$1
   local changed=`stat -c %Y "$filename"`
   local now=`date +%s`
   local elapsed
   let elapsed=now-changed
   echo $elapsed
}

/root/pki/fetch.sh
# push new certs if less that 7.5d old
if [ $(age "/root/pki/cert.pem") -lt 648000 ]; then
        /root/pki/push.sh
fi


fetch.sh

Code Select Expand


#!/bin/sh
rsync --checksum -Ltve 'ssh' root@192.168.xx.yy:/etc/letsencrypt/live/[mydomain]/* /root/pki/


push.se

Code Select Expand


#!/bin/sh
echo Pushing config
scp /root/pki/*.pem root@192.168.aaa.bbb:/etc/pki/tls/letsencrypt/
scp /root/pki/*.pem root@192.168.ccc.ddd:/etc/pki/tls/letsencrypt/
[...]
echo Restarting remote services
ssh root@192.168.aaa.bbb "systemctl restart postfix"
ssh root@192.168.aaa.bbb "systemctl restart dovecot"
ssh root@192.168.ccc.ddd "systemctl restart apache2"
[...]


192.168.xx.yy is the web server that also runs the LetsEncrypt certbots hence generates certificates
192.168.aaa.bbb is an email server
192.168.ccc.ddd is another web server

I had a similar problem. LetsEncrypt cert bots running on public web server, every time certs being refreshed they need to be distributed around a few more servers (another web server, a smtp server, etc). I solved it by writing a small script which runs weekly, checks the last-modified date on the cert files, and if less than a week old redistributes them to the other servers and remotely restarts the relevant services to pick up the new certs. Was pretty easy to do with a combination of ssh and scp, at least in an all-unix environment. I can paste the script here later if of interest.

Ok so... I've continue to look at this and haven't found the answer but have narrowed it down.

First of all the rejects from the logs appear to be a red herring and basically this: https://forum.pfsense.org/index.php?topic=39960.0.

I've further noticed that it's only one particular host which is problematic namely my FreeNAS server. Troubleshooting with iperf3, I can iperf from a MacBook Pro and from Windows 10 to other hosts (Linux VMs) on the same subnet and across subnets (via opnsense) in all cases consistently at around 940MBit/s, so as expected.

The FreeNAS server has two physical NICs, currently one on LAN and one on DMZ. When I iperf3 to the same subnet I get the same consistent 940MBit. However when I cross subnets - from LAN to DMZ or the other way around (for testing) I get something like this:

Code Select Expand

foo$ ./iperf3 -c 192.168.200.10
Connecting to host 192.168.200.10, port 5201
[  4] local 192.168.1.174 port 49559 connected to 192.168.200.10 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec   210 KBytes  1.71 Mbits/sec                 
[  4]   1.01-2.01   sec  0.00 Bytes  0.00 bits/sec                 
[  4]   2.01-3.01   sec  0.00 Bytes  0.00 bits/sec                 
[  4]   3.01-4.00   sec  0.00 Bytes  0.00 bits/sec                 
[  4]   4.00-5.01   sec  0.00 Bytes  0.00 bits/sec                 
[  4]   5.01-6.00   sec  0.00 Bytes  0.00 bits/sec                 
[  4]   6.00-7.01   sec  0.00 Bytes  0.00 bits/sec                 
[  4]   7.01-8.00   sec  0.00 Bytes  0.00 bits/sec                 
[  4]   8.00-9.01   sec  0.00 Bytes  0.00 bits/sec                 
[  4]   9.01-10.00  sec  0.00 Bytes  0.00 bits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   210 KBytes   172 Kbits/sec                  sender
[  4]   0.00-10.00  sec  65.0 KBytes  53.3 Kbits/sec                  receiver

Not good! Something along the way, or the FreeNAS server itself, is throwing away packages. There is nothing in particular that I can see in the opnsense nor in the FreeNAS logs, and I don't really know how to troubleshoot this.

FreeNAS runs on FreeBSD 11.1.

Help...?!