Will it 10Gb/s OPNsense?

[PhiloEpisteme]

Hi folks, I'm new to the forums, OPNsense, and networking in general. Please forgive any terminology mistakes or misunderstandings on my part.

I am looking to add an OPNsense box to my SOHO network to use its firewall capabilities and to split my home network into at least 4 VLANS (5 if you count the guest network) and hopefully be able to achieve 10Gbs LAN speeds within and between VLANS.

VLAN1: Work computer(s)
VLAN2: FreeNAS servers
VLAN3: Personal computers and phones
VLAN4: IoT devices such as television, speakers, lights, etc.

Key Network Uses
If possible, I'd like a reliable 10Gb/s connection between VLAN1 and VLAN2.

I'd like to use a personal computer in VLAN3 to be able to access devices in other VLANs such as my FreeNAS server or one of my IoT devices.

I'd like to split my wireless devices across multiple VLANs, for example laptops and cellphone belong together but IoT devices belong in another VLAN.

I imagine my use case is not that extraordinary. If so, what kind of hardware am I looking at? The piece I am specifically worried about it getting a near 11Gb/s speed between say my work computer in VLAN1 to my FreeNAS machine in VLAN2. Many of the other devices either don't need bandwidth that high or are wireless anyway so I am less concerned about the performance there.

As an added bonus, my current situation is such that my FreeNAS machines are directly connected to one another via 10Gb/s fibre to make backups between them significantly faster. Is there any way to expose my FreeNAS machines to the rest of my network via 10Gb/s links using OPNsense or would I have to use a 10Gb/s L2 switch between the FreeNAS machines and OPNsense and put all 10Gb/s devices on that switch in the same VLAN?

I did some research on hardware and performance and it seems that if I am genuinely interested in 10Gb/s performance I'll likely need to build something myself rather than rely on the all-in-one mini-pc solutions lots of folks use.

https://calomel.org/network_performance.html
https://calomel.org/freebsd_network_tuning.html

I have some hardware around the house I am happy to use but am also happy to build another machine or purchase an off-the-shelf solution if it is appropriate.

What I own

• ASUS ROG Strix H370-IASUS ROG Strix H370-I
  
  • Intel® I219V, 1 x Gigabit LAN Controller Dual interconnect between the Integrated Media Access Controller (MAC) and Physical Layer (PHY)
  • Realtek® RTL8111H, 1 x Gigabit LAN
  • 1x PCIe3.0x16
• Intel Core i5-8400 6-core 2.8GHz 8 GT/s bus speed
• G.SKILL Ripjaws V 16GB (2 x 8GB) DDR4 SDRAM DDR4 2666

With the above hardware I'm a bit limited. If I pick up a 10Gbs NIC I'll only have 2 1Gbs NICs left, and one of those is that Realtek NIC. Perhaps I'm just looking for confirmation here but with as many devices as I'd like to connect it'll likely be that I need a board with more onboard NICs (so long as they don't offload too much work the cpu) and/or 2+ 8x PCIe 3.0 slots.

Thanks for your time. I've done some searching already and have found a few useful links but clearly I still have questions. If I am just bad at searching feel free to throw a link at me. Any advice or accessible reading would be greatly appreciated.

[mimugmail]

If you dont do Nat on 10g links throuput is not a problem

[PhiloEpisteme]

If you dont do Nat on 10g links throuput is not a problem

What is the consequence of disabling NAT? So long as all machines on the VLANs can access the web I'm happy.

When you say 10Gb/s throughput is no problem I imagine this assumes appropriate hardware. There are precious few mini PCs with multiple 10G links. Is it foolish to go the Supermicro mini-ITX build route?

[mimugmail]

I'm not a hardware specialist, usually I choose from Thomas Krenn as they have tested devices, also with super Micro boards. When you Nat on WAN you are limited to the cpu. Vlan to Vlan without Nat is nearly wirespeed.

[PhiloEpisteme]

When you Nat on WAN you are limited to the cpu. Vlan to Vlan without Nat is nearly wirespeed.

Thanks for the advice. As I'm a bit new would you mind expanding a little bit? Am I correct that I have to use NAT on WAN in order to give all of my machines access to the internet, yes?

As far as disabling NAT for VLAN to VLAN, what feature am I losing by doing that? What is the benefit to enabling NAT between VLANs in a setup like mine?

[mimugmail]

To wan you need Nat, indeed. So when you have 10g WAN you will not achieve such rates (with your hardware). Internal routing usually is never natted, so this is perfectly ok.

[PhiloEpisteme]

Thanks for the input!

So I've done some research into hardware. Ideally I'd like 2 10G ports in the event I end up with 2 10G VLANs and want a 10G connection between them. I'd also like 8 1G ports. Lacking any L2 switches my current system would use 6 ports to start.

Motherboard
I'm looking at the AMD EPYC series processors for this; specifically in the following Supermicro board.

• M11SDV-4CT-LN4F uses the AMD EPYC 3101 2.1GHz (2.9GHz max) 4 cores 8 threads @35W
• M11SDV-4C-LN4F uses the AMD EPYC 3151 2.7GHz (2.9GHz max) 4 cores 8 threads @45W
• M11SDV-8CT-LN4F uses the AMD EPYC 3201 2.5GHz (1.5GHz max) 8 cores 8 threads @30W
• M11SDV-8C-LN4F uses the AMD EPYC 3251 2.5GHz (3.1GHz max) 8 cores 16 threads @55W

They all have 1x PCIe3.0x16 slot which supports port bifurcation to support adding on 2 additional PCIe3.0x8 cards. I'm currently looking at one of the following

NICs
From Motherboard: i350-AM4 (4x 1G)
10G via PCIe: One of

• Chelsio T420-CR (2x 10G)
• Chelsio T440-CR (4x 10G)
• Chelsio T422-CR (2x 10G + 2x 1G)

1G via PCIe: i350-T4V2 (4x 1G)

At a minimum this configuration would give me 8x 1G + 2x 10G
At a maximum 10x 1G + 2x 10G or 8x 1G + 4x 10G.

Memory
I'd pick up 2x4GB ECC ram modules.

Do you suggest any of these boards for the configuration above? I'm leaning toward the AMD EPYC 3151 thinking that the 2.7 4-core 8thread design at 45W is a good sweet spot. Will it be able to handle the load assuming no NAT between VLANS and only NAT to WAN?

Finally, am I making a huge mistake by going with a SoC? Would it be better to pick up one of the socket boards to be able to upgrade the CPU in the future?

I realize replying here may make folks less likely to see it since this is a thread already in action. If I get little attention to this perhaps I'll post the specs above in a new thread.

[marshalleq]

I was very tempted to go and buy a netgate appliance tonight - however, i thought I'd do a quick google on the state of 10G internet in NZ and apparently we're getting consumer 10G next year.  Also, it's apparently not going to be much more expensive than our 1G.  So maybe, I should hold off.  We're supposedly getting a 2,4 and 8 G option as well with a 25G option coming sometime later.  Sounds crazy.

If it's cheap enough I might get it - last time they just upgraded all us top tier 100M peeps to 1G for free so, who knows maybe they'll do the same.  All in all thanks to Unbundling of our exchanges, internet here is a real success story now.