Small formfactor router

Started by Simaryp, August 05, 2025, 10:08:34 PM

Previous topic - Next topic
I went a bit wild with VLAns, like my printer has it's own VLAN, my android and media devices have their own, and my server and linux pcs etc. And then I have roules to allow traffic to the server and so on.

But if I put all myprivate devices into one, all the traffic goes via the switch.

My internet connection is only 150Mbit/s down and 40 up. So I don't need that for net, only like if I would have parallel traffic between VLANS I would need more than 1G between switch and FW.

For the WiFi stuff I think the radio us limiting anyway. I have a EAP245 and I only get around 300Mbit between my laptop and server. I guess I could upgrade the AP, but I don't know if newer ones would bring any real world benefit.

I heard on Youtube that the fan device can become quite loud. So I thought a passive one with a noctua might work.

Something like this would be nice, passive and low power, but the price
https://shop.opnsense.com/product/dec750-opnsense-desktop-security-appliance/

Or maybe I should get something like this
https://www.kleinanzeigen.de/s-anzeige/lenovo-m720q-tiny-pc-intel-core-i3-8gb-ram-256gb-ssd/3155678802-278-1744

and put my NIC in there and maybe upgrade to SFP+ when needed.

As I said, there is mostly inter-VLAN traffic from your main LAN to any other VLAN, even if you went wild with that. So, you can combine all other VLANs on one interface and LAN on the other. Plus, the 4 or 6 NIC boxes are not gettings as hot as the ones with SFP+ adapters and are cheaper, too.
Intel N100, 4* I226-V, 2* 82559, 16 GByte, 500 GByte NVME, ZTE F6005

1100 down / 800 up, Bufferbloat A+

You mean making one cable for main VLAN from switch to FW and then a second one with a trunc of all other VLANs? Thats maybe a better idea than havjng two combined via link aggregation and having everything via it.

So maybe the 4Port N150 passive is the way to go.

Yes, especially considering that a LAGG does not work in practive for home setups. That is because any IP stream between two machines can only use one connection at a time. Most switches can only distribute packets by MAC, IP or port number, round robin exists only for Infiniband.

The best thing you could have is to have multiple streams between two machines, but since most switches cannot even do port distribution, it will not help, either.

And since most switches just distribute based on MAC, you can end up in situations where most or all of your machines communicate over the same link, while the other is scarcely used. LAGG functions only for large installations, where statistics play in your favor.

By dividing up the links into the probable sides of a communication, you can manually select that both links are used for inter-VLAN communications.
Intel N100, 4* I226-V, 2* 82559, 16 GByte, 500 GByte NVME, ZTE F6005

1100 down / 800 up, Bufferbloat A+

Thanks, that was helpfull information.

I think I can also ignore the 2.5G of my new server for the moment. I am using my laptop primarily, which over WiFi only gets 300Mbit/s and other connections are mostly strewaming, which is also not really taking lot's of bandwidth.

So I think I will go for the passive 4 Nic version with N150.

Is the RAM and SSD included goodor should I shop seperately?

I always shop separately, because it is cheaper and you know exactly what you get. Pay attention to buy an enterprise class NVME with high TBW, because RRD and Netflow on ZFS will eat through it like a hot knife through butter.
Intel N100, 4* I226-V, 2* 82559, 16 GByte, 500 GByte NVME, ZTE F6005

1100 down / 800 up, Bufferbloat A+

August 11, 2025, 04:06:20 PM #37 Last Edit: August 11, 2025, 05:15:25 PM by Simaryp
Do you have a recommendation for a specific model?

Maybe this Gigabyte one. It has a TBW of 110 TB.
GP-GSM2NE3128GNTD

I use two types NVMEs for my PRX and OPNsense

Samsung 980 500GB - 300 TBW
Lexar SSD NM790 512GB - 500 TBW

Regards,
S.
Networking is love. You may hate it, but in the end, you always come back to it.

OPNSense HW
APU2D2 - deceased
N5105 - i226-V | Patriot 2x8G 3200 DDR4 | L 790 512G - VM HA(SOON)
N100   - i226-V | Crucial 16G  4800 DDR5 | S 980 500G - PROD

August 11, 2025, 10:06:25 PM #39 Last Edit: August 11, 2025, 10:13:16 PM by meyergru
I would use a type that has at least 500 TBW, independent on capacity. Preferably, you could use a larger capacity, which you do not need specifically, but which gives you more headroom for writing. I always use specimens with real RAM cache for obvious reasons (i.e. SLC cache does not help).

On the other hand, speed does not matter at all or is even detrimental, because newer PCIE 4.0 or 5.0 NVMEs tend to get much hotter without any visible benefit.

You can search for the parameters on many product search sites, like this: https://geizhals.de/?cat=hdssd&xf=7525_M.2+(PCIe)

For these types of application, a Transcend MTE220S might be a good choice, but there are others.

That being said, I made a bad decision for my last N100 box and chose a 500 GByte Kioxia Exeria G2 because of its low price. It only has a 200 TBW rating (or 400 TBW per TB).

Here is the smartctl output for that drive:

smartctl 7.5 2025-04-30 r5714 [FreeBSD 14.3-RELEASE-p1 amd64] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       KIOXIA-EXCERIA G2 SSD
Serial Number:                      44OA40XTK71S
Firmware Version:                   ECFA17.3
PCI Vendor/Subsystem ID:            0x1e0f
IEEE OUI Identifier:                0x8ce38e
Total NVM Capacity:                 500,107,862,016 [500 GB]
Unallocated NVM Capacity:           0
Controller ID:                      1
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          500,107,862,016 [500 GB]
Namespace 1 Formatted LBA Size:     4096
Namespace 1 IEEE EUI-64:            8ce38e 0300993420
Local Time is:                      Mon Aug 11 21:59:09 2025 CEST
Firmware Updates (0x12):            1 Slot, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x0a):         Cmd_Eff_Lg Telmtry_Lg
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     72 Celsius
Critical Comp. Temp. Threshold:     90 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     7.69W       -        -    0  0  0  0        1       1
 1 +     6.18W       -        -    1  1  1  1        1       1
 2 +     5.42W       -        -    2  2  2  2        1       1
 3 -   0.0500W       -        -    3  3  3  3     7000    5000
 4 -   0.0050W       -        -    4  4  4  4    13000   36000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 -     512       0         2
 1 +    4096       0         1

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:                   0x00
Temperature:                        58 Celsius
Available Spare:                    100%
Available Spare Threshold:          5%
Percentage Used:                    12%
Data Units Read:                    3,532,700 [1.80 TB]
Data Units Written:                 34,666,845 [17.7 TB]
Host Read Commands:                 79,386,577
Host Write Commands:                266,663,545
Controller Busy Time:               560
Power Cycles:                       11
Power On Hours:                     3,024
Unsafe Shutdowns:                   4
Media and Data Integrity Errors:    0
Error Information Log Entries:      80
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Thermal Temp. 1 Transition Count:   100
Thermal Temp. 1 Total Time:         18823

Error Information (NVMe Log 0x01, 16 of 63 entries)
No Errors Logged

Self-test Log (NVMe Log 0x06, NSID 0xffffffff)
Self-test status: No self-test in progress
No Self-tests Logged

Notice how this drive has only 3024 hours of use (i.e. 126 days or 4 months), yet 12% or 17.7 TByte of its life has already been eaten by RRD and Netflow. At that rate, it will last much less than 3 years in total, which is probably less than the expected life of the box itself.
Intel N100, 4* I226-V, 2* 82559, 16 GByte, 500 GByte NVME, ZTE F6005

1100 down / 800 up, Bufferbloat A+

The Addlink D60 is a 1DWPD device (5Y warranty). Not very impressive, but more than most. It's also a burning weenie roaster, like every other M.2 SSD I've used. But if you're stuck with M.2, it's an option. There are few 3DWPD M.2 SSDs left - maybe Micron's 7300 or 7450 Max (not Pro).

As a data point, I use basic rules with logging enabled, Kea, NTP, RRD, no IPS, no netflow. Static IP Internet service with servers, so ~300 sessions average (at the moment - it's been higher). After 8 months, SMART indicates:

Data Units Read:                    64,626 [33.0 GB]
Data Units Written:                1,305,190 [668 GB]

A bit less than 18TB. But then I have a 1.6TB 3DWPD device, so 18TB would be a couple weeks' worth of writes.

With a 10G fiber cable, you can use just that one cable between fw and the switch. "Trunk" as you will, more specifically .1q.
Then on switch you carve out your vlans (lan, lan2, lan3, wan, etc etc). Unless you do span all the time (ids, other), just need 1port for wan (isp).

A one-port SFP+ device is technically all that's needed if you have a decent managed switch.

I not sure what was meant by "vlans with routes". Did you build L3 vlans, and if so why? Just need L2 vlans and .1q to fw.

August 12, 2025, 06:35:27 AM #42 Last Edit: August 12, 2025, 07:00:16 AM by BrandyWine
Quote from: Simaryp on August 11, 2025, 04:06:20 PMDo you have a recommendation for a specific model?

Maybe this Gigabyte one. It has a TBW of 110 TB.
GP-GSM2NE3128GNTD
Get the sizes you need
150 TBW https://www.amazon.com/dp/B07ZGK3K4V , noted: the Samsung 980 is way better, also about 3x the price.
Crucial https://www.amazon.com/dp/B09S2MN8JH

Get what you can out of zfs, noatime



I am familar with Geizhals.

Then maybe the 500GB Transcend MTE220S it is. And some 8GB Kingston module.

But first project will be a new no HDD server suitable for living room and then refresh of the network.

@BrandyWine. But what will be the power consumption of this 1 SFP+ fiber compared to the two RJ ports?

I know about this router on a stick model, but I had enough portsso far to use a dedicated WAN on the FW.

August 12, 2025, 08:24:41 AM #44 Last Edit: August 12, 2025, 08:32:21 AM by meyergru
@Brandywine: Objection on all parts:

1. I use a 10G DAC connection as well - unless you have a specific need for an inter-VLAN or internet connection that actually uses the full speed of this, it is a waste of money and energy. Also, the NICs make the box much hotter (all of the 10G models have active cooling and deservedly so) and 10G switches are much more expensive, too.
As for the need of that speed: remember, that even a huge RAID NAS has a write speed of one HDD effectively, i.e. ~200 MByte/s or 2 GBps, which is well below what you can achieve with 2.5 Gbps NICs. So, only if your client(s) has 10G and your NAS uses SSD storage, you will gain almost nothing by using 10G. Been there, done that (actually, doing it now).

2. IMHO, 150 TBW is way too small if you want RRD and Netflow, as I already demonstrated. Notice my drive has 200 TBW and will be gone after < 3 years for home usage.

3. noatime is the default on OpnSense anyway, so there is nothing to set at all. Also, noatime - which means "do not change any metadate on access (i.e. reads)" - will do next nothing for a COW filesystem like ZFS, where the rewrites on statistics data already copy all of the written sectors.
Intel N100, 4* I226-V, 2* 82559, 16 GByte, 500 GByte NVME, ZTE F6005

1100 down / 800 up, Bufferbloat A+