3 Gb/s fiber, 1Gb/s LAN

[dvdwsn]

I've been thinking about testing out OPNsense for a while. I'm currently on Untangle (Arista), and although I'm satisfied, there are more and more features I'm wanting that it doesn't have.
One feature is Link Aggregation. I have a new ISP, and they are providing me with a 3Gb/s symmetrical connection. Unfortunately my network is entirely 1Gb/s, even the switch only has SFP ports, although my gateway server does have 1x 10Gb/s port. 
What I'm wondering is, can I use the 10Gb/s NIC on my router/gateway server to connect to the ISP's modem, then use Link Aggregation from my server which has 4x 1Gb/s ports, to connect to my switch, which also supports Link Aggregation (802.3ad)?

[CJ]

You should be able to, but you'll still only get 1G max on a single connection.  Are you planning on upgrading to multigig in the near future?  Unless 3G is the same price or cheaper than 1G I'm not sure I'd bother.  Most people don't even fully use a 1G connection.

[dvdwsn]

you'll still only get 1G max on a single connection.  Are you planning on upgrading to multigig in the near future?  Unless 3G is the same price or cheaper than 1G I'm not sure I'd bother.

- Yeah I read that somewhere, and if I understand correctly, any individual data transfer, say a http download, will max out at 1G. However, another client on the network can access another of the 1G connections if their total download speed exceeds what's left in the first 1G connection?
- I am looking at a new switch that has 2.5G ethernet ports and some 10G SFP+ ports.
- 3G came out to $5 more /month than 1.5G. The other difference was 3G is symmetrical, and the 1.5G was much slower up.


The 10G card in my gateway server has 2 ports. If I were to buy a 10G card for my other main server, how bad of an idea would it be to create a 2nd LAN on the gateway? I think that would cause the gateway to take on switching overhead for traffic across it's NIC and possibly hammer the CPU?

[CJ]

- Yeah I read that somewhere, and if I understand correctly, any individual data transfer, say a http download, will max out at 1G. However, another client on the network can access another of the 1G connections if their total download speed exceeds what's left in the first 1G connection?

Yes, that is how it should work.

- I am looking at a new switch that has 2.5G ethernet ports and some 10G SFP+ ports.
- 3G came out to $5 more /month than 1.5G. The other difference was 3G is symmetrical, and the 1.5G was much slower up.

Nice.  I retract my former comment.

The 10G card in my gateway server has 2 ports. If I were to buy a 10G card for my other main server, how bad of an idea would it be to create a 2nd LAN on the gateway? I think that would cause the gateway to take on switching overhead for traffic across it's NIC and possibly hammer the CPU?

I'm a bit unclear as to what you're asking.  Can you create a diagram showing what you're considering?

[dvdwsn]

I'm a bit unclear as to what you're asking.  Can you create a diagram showing what you're considering?

I've attached a diagram below. The file server isn't connected to the switch, so traffic will traverse the NIC on the gateway. Does this create any performance issues?

[CJ]

Ah, I understand now, you want to bridge the OPNsense 10g NICs.  Yes, attempting to do 10g storage traffic across OPNsense would introduce a performance hit, but it might still be faster than 1g depending on your hardware.

If you're able to, you might be able to rearrange things so that you can do a direct 10g connection from your PC to the NAS.  That would give you faster performance for that at least.  There are some cheaper options for 10g switches but it can still be pricey to start.

One thing to keep in mind is the performance of your PC and NAS.  Are they powerful enough to actually take advantage of 10g?  You may be spending a chunk of change for not much return.