Messages

I posted recently about an upcoming build. I'm still deciding on hardware etc and feel no closer to selecting a board than I was a week ago. Nonetheless, I would like to try to clear up something regarding the difference between onboard controllers vs NICs.

What I am specifically curious about is whether, in high throughput environments, there is a real disadvantage to using the onboard interfaces vs the NICs assuming quality controllers.

For example, consider a situation where one has a 4-port board such as the Supermicro M11SDV-4C-LN4F which has 4x 1GbE with Intel I350-AM4 with a PCIe2.0 link per the ark.intel.com pages and pair that with an Intel I350-T4V2 4x 1GbE card which uses a PCIe2.1 link. Would one expect these onboard controllers to perform comparable to the NIC given that they are both quality controllers? Or will there be some performance penalty due to perhaps the onboard controllers making more use of the CPU, being slow to get data between the onboard controller and the PCIe bus to the NIC, or some other reason?

If there is no serious performance penalties (assuming one purchases a board with quality onboard controllers) then I will likely consider one of these 2-4 port boards with i350 controllers and add as many additional NICs as required to bring my port count up.

[Motherboard]

Thanks for the input!

So I've done some research into hardware. Ideally I'd like 2 10G ports in the event I end up with 2 10G VLANs and want a 10G connection between them. I'd also like 8 1G ports. Lacking any L2 switches my current system would use 6 ports to start.

Motherboard
I'm looking at the AMD EPYC series processors for this; specifically in the following Supermicro board.

• M11SDV-4CT-LN4F uses the AMD EPYC 3101 2.1GHz (2.9GHz max) 4 cores 8 threads @35W
• M11SDV-4C-LN4F uses the AMD EPYC 3151 2.7GHz (2.9GHz max) 4 cores 8 threads @45W
• M11SDV-8CT-LN4F uses the AMD EPYC 3201 2.5GHz (1.5GHz max) 8 cores 8 threads @30W
• M11SDV-8C-LN4F uses the AMD EPYC 3251 2.5GHz (3.1GHz max) 8 cores 16 threads @55W

They all have 1x PCIe3.0x16 slot which supports port bifurcation to support adding on 2 additional PCIe3.0x8 cards. I'm currently looking at one of the following

NICs
From Motherboard: i350-AM4 (4x 1G)
10G via PCIe: One of

• Chelsio T420-CR (2x 10G)
• Chelsio T440-CR (4x 10G)
• Chelsio T422-CR (2x 10G + 2x 1G)

1G via PCIe: i350-T4V2 (4x 1G)

At a minimum this configuration would give me 8x 1G + 2x 10G
At a maximum 10x 1G + 2x 10G or 8x 1G + 4x 10G.

Memory
I'd pick up 2x4GB ECC ram modules.

Do you suggest any of these boards for the configuration above? I'm leaning toward the AMD EPYC 3151 thinking that the 2.7 4-core 8thread design at 45W is a good sweet spot. Will it be able to handle the load assuming no NAT between VLANS and only NAT to WAN?

Finally, am I making a huge mistake by going with a SoC? Would it be better to pick up one of the socket boards to be able to upgrade the CPU in the future?

I realize replying here may make folks less likely to see it since this is a thread already in action. If I get little attention to this perhaps I'll post the specs above in a new thread.

When you Nat on WAN you are limited to the cpu. Vlan to Vlan without Nat is nearly wirespeed.

Thanks for the advice. As I'm a bit new would you mind expanding a little bit? Am I correct that I have to use NAT on WAN in order to give all of my machines access to the internet, yes?

As far as disabling NAT for VLAN to VLAN, what feature am I losing by doing that? What is the benefit to enabling NAT between VLANs in a setup like mine?

If you dont do Nat on 10g links throuput is not a problem :)

What is the consequence of disabling NAT? So long as all machines on the VLANs can access the web I'm happy.

When you say 10Gb/s throughput is no problem I imagine this assumes appropriate hardware. There are precious few mini PCs with multiple 10G links. Is it foolish to go the Supermicro mini-ITX build route?

Hi folks, I'm new to the forums, OPNsense, and networking in general. Please forgive any terminology mistakes or misunderstandings on my part.

I am looking to add an OPNsense box to my SOHO network to use its firewall capabilities and to split my home network into at least 4 VLANS (5 if you count the guest network) and hopefully be able to achieve 10Gbs LAN speeds within and between VLANS.

VLAN1: Work computer(s)
VLAN2: FreeNAS servers
VLAN3: Personal computers and phones
VLAN4: IoT devices such as television, speakers, lights, etc.

Key Network Uses
If possible, I'd like a reliable 10Gb/s connection between VLAN1 and VLAN2.

I'd like to use a personal computer in VLAN3 to be able to access devices in other VLANs such as my FreeNAS server or one of my IoT devices.

I'd like to split my wireless devices across multiple VLANs, for example laptops and cellphone belong together but IoT devices belong in another VLAN.

I imagine my use case is not that extraordinary. If so, what kind of hardware am I looking at? The piece I am specifically worried about it getting a near 11Gb/s speed between say my work computer in VLAN1 to my FreeNAS machine in VLAN2. Many of the other devices either don't need bandwidth that high or are wireless anyway so I am less concerned about the performance there.

As an added bonus, my current situation is such that my FreeNAS machines are directly connected to one another via 10Gb/s fibre to make backups between them significantly faster. Is there any way to expose my FreeNAS machines to the rest of my network via 10Gb/s links using OPNsense or would I have to use a 10Gb/s L2 switch between the FreeNAS machines and OPNsense and put all 10Gb/s devices on that switch in the same VLAN?

I did some research on hardware and performance and it seems that if I am genuinely interested in 10Gb/s performance I'll likely need to build something myself rather than rely on the all-in-one mini-pc solutions lots of folks use.

https://calomel.org/network_performance.html
https://calomel.org/freebsd_network_tuning.html

I have some hardware around the house I am happy to use but am also happy to build another machine or purchase an off-the-shelf solution if it is appropriate.

What I own

• ASUS ROG Strix H370-IASUS ROG Strix H370-I
  
  
  
  • Intel® I219V, 1 x Gigabit LAN Controller Dual interconnect between the Integrated Media Access Controller (MAC) and Physical Layer (PHY)
  • Realtek® RTL8111H, 1 x Gigabit LAN
  • 1x PCIe3.0x16
• Intel Core i5-8400 6-core 2.8GHz 8 GT/s bus speed
• G.SKILL Ripjaws V 16GB (2 x 8GB) DDR4 SDRAM DDR4 2666

With the above hardware I'm a bit limited. If I pick up a 10Gbs NIC I'll only have 2 1Gbs NICs left, and one of those is that Realtek NIC. Perhaps I'm just looking for confirmation here but with as many devices as I'd like to connect it'll likely be that I need a board with more onboard NICs (so long as they don't offload too much work the cpu) and/or 2+ 8x PCIe 3.0 slots.

Thanks for your time. I've done some searching already and have found a few useful links but clearly I still have questions. If I am just bad at searching feel free to throw a link at me. Any advice or accessible reading would be greatly appreciated.