Network setup with OPNsense for temporary event

[Snoopy913]

Hello all,

I wanted some opinions and feedback on a budget friendly network setup for a student design competition I am currently planning. The Event is every year, so this will be a long term investment. The competition is for designing a racecar and doing dynamic tests with these cars.

The goal for the network is to provide the infrastructure for a livestream to youtube, uplink for a timekeeping publishing to a VPS, local network for light control over Wifi and internet over Wifi for the volunteers.

The location is the main problem, there is no infrastructure available there so all our setup is temporary for 7 days, it is a retired airstrip about 40m (~130ft) wide and 500m (~1600m) long with both sites grass where the equipment is positioned. The starting point is at one end of the airstrip where we have power and where we want to position the Router, Switch, Livestreaming PC,...).

As an Uplink we want to use a mobile Modem for all basics (Wifi for the volunteers, timekeeping) and a Starlink Antenna for the Livestream to Youtube (we have observed that due to the increasing number of students and visitors at the same place, the mobile network is to crowded for livestreaming).

As an potential future project we could lay a fiber cable, but that will have to be 300m and routed under a train track, so we don't want to plan with it.

In total that would make 3 WAN connections for OPNsense and currently 3 partly separate networks. The OPNsense router + Switch + Livestream PC should all be Rack Mountable and stored in a Rack Flightcase for easy storage and transport.

This year we had a good test using Media Boxes with a 2,5Gbit Switch Power and a MiniPC for creating a video stream from the cameras and sending everything to the main Livestream PC. These Media Boxes have an Ethernet and Power Inlet and Outlet and will be connected to each other, with distances of ~80m, but we might change this distance to 50m to be more flexible with the camera placement.

Attached is the rough network drawing, which shows the first 2 media Boxes on each side of the airstrip, there will be 5-10 Boxes in total on each side, depending on the distance between the boxes. There will be only 4-5 Boxes with a MiniPC and camera connected, the others will only have a switch and maybe an AP.

I am trying to make sure that everything in the infrastructure at least 2,5Gbit and I was going with normal Ethernet CAT6.A cables because I need to route Power none the less and the distance between the boxes will mostly not exceed 100m so copper should be fine, so fiber should not bring any advantage.

The Video streams will have not more then 10Mbit but there is a protocol called NDI which we might use in the future and this could require a bandwidth of 100Mbit per 1080p stream, so the baseload of the network to the Livestreaming PC could exceed 500Mbit.

Now to the Hardware:
I could use Ubiquiti for everything but especially for the router with 3 WAN and at least one 2,5Gbit LAN connection this is unnecessary expensive, so I wanted to go with some aliexpress router with i226 NICS like:
https://de.aliexpress.com/item/1005006468199705.html
As I have some DDR4 SO-DIMMs already.

For the Media Boxes we already have some Switches from this year: https://www.reichelt.de/ch/de/shop/produkt/switch_5-port_gigabit_ethernet-422777
For newly build Media Boxes, I would switch to PoE++ Switches, to make it easy to connect APs

For the Main Switch and Wifi APs I would like to use something managed via the same configuration tool to make it more easy.

The be double sure that I am not creating a bottleneck with the OPNsense router, only the traffic going to "the internet" and traffic between my 3 separated networks is going through OPNsense, all local should have no need to be routed through OPNsense, right? In this case I don't need a very powerful OPNsense device.

Another question for me is, I am not yet sure how I separate the three networks, but most likely setting different vLANs for different Wifi SSIDs. Can I define in OPNsense which vLAN is using which WAN connection? I want to make sure that the livestream has the highest priority.

If someone has a recommendations for good PoE outdoor APs for large areas, this would also be welcome.

If someone has any doubts that this will work, or improvements, that is very welcome.

[nero355]

In total that would make 3 WAN connections for OPNsense and currently 3 partly separate networks.

Can I define in OPNsense which vLAN is using which WAN connection? I want to make sure that the livestream has the highest priority.

IMHO creating that setup is just annoying from all the topics that I have seen so far with Multi-WAN setups so I would use 3 seperate OPNsense Routers if your budget allows it :)

The Video streams will have not more then 10Mbit but there is a protocol called NDI which we might use in the future and this could require a bandwidth of 100Mbit per 1080p stream, so the baseload of the network to the Livestreaming PC could exceed 500Mbit.

Why such a high bitrate ?!

IIRC 1080p Blu-Ray doesn't even exceed 42 Mbps or so...

And with all the new codecs that came after that I am betting you could go even lower with the same quality! :)

But maybe your NDI protocol changes all of this ofcourse ?

If someone has a recommendations for good PoE outdoor APs for large areas, this would also be welcome.

I guess that brings us very quickly back to Ubiquiti Outdoor UAPs if you want to keep things relatively cheap ?! ^_^



Good luck! with the project and let us know where to watch this when you all done with the setup ;)

[BrandyWine]

How does the video get from vehicle to the livestream PC?
What's the internet link, your 5GLTE device? If everyone else is using the same cell then bandwidth could/might be impacted.

Anything streaming to a utoob channel, you can buy a small device from amzon or the like that takes in video (hdmi, composite, ntsc/pal), hardware encodes it, and then streams it to the utoob channel. So basically no work for that part, dont need a PC. Use the one that has eth port so no reliance on wifi.

Wifi for the team, you just need a wifi AP. What are they doing on the wifi (internet, other)?

[Snoopy913]

In total that would make 3 WAN connections for OPNsense and currently 3 partly separate networks.

Can I define in OPNsense which vLAN is using which WAN connection? I want to make sure that the livestream has the highest priority.

IMHO creating that setup is just annoying from all the topics that I have seen so far with Multi-WAN setups so I would use 3 seperate OPNsense Routers if your budget allows it :)

The Video streams will have not more then 10Mbit but there is a protocol called NDI which we might use in the future and this could require a bandwidth of 100Mbit per 1080p stream, so the baseload of the network to the Livestreaming PC could exceed 500Mbit.

Why such a high bitrate ?!

IIRC 1080p Blu-Ray doesn't even exceed 42 Mbps or so...

And with all the new codecs that came after that I am betting you could go even lower with the same quality! :)

But maybe your NDI protocol changes all of this ofcourse ?

If someone has a recommendations for good PoE outdoor APs for large areas, this would also be welcome.

I guess that brings us very quickly back to Ubiquiti Outdoor UAPs if you want to keep things relatively cheap ?! ^_^



Good luck! with the project and let us know where to watch this when you all done with the setup ;)

Yes it's basicaly the NDI protocol which skyrockets the bandwidth, that's why I want to avoid it. If I can use some other protocol, Like SRT, we will be in the 10Mbps area.

I started planning with U7 Outdoor, but I have not yet found any tests on the what actual area coverage I can expect in the open field.

This was our first trial with the Livestream a 2 weeks ago, it started pretty bad, but got better in the end and we now know the areas we have to improve, like Camera and network:
https://youtube.com/playlist?list=PLJ-leheBA7Kg&si=0uBTGvQExG0eVx-8

In about a year, same channel, there will be our next possibility to Test the improvements.

[Snoopy913]

How does the video get from vehicle to the livestream PC?
What's the internet link, your 5GLTE device? If everyone else is using the same cell then bandwidth could/might be impacted.

Anything streaming to a utoob channel, you can buy a small device from amzon or the like that takes in video (hdmi, composite, ntsc/pal), hardware encodes it, and then streams it to the utoob channel. So basically no work for that part, dont need a PC. Use the one that has eth port so no reliance on wifi.

Wifi for the team, you just need a wifi AP. What are they doing on the wifi (internet, other)?

Yes the 5G mobile Network cannot be used, as soon as some people are there, IT IS to crowded for streams.

We would Stream the in vehicle cameras via WiFi, for that the APs would be used. I currently just don't know which distance between the APs I can have to have a good coverage over the complete Track.

We will most likely have 4-5 fixed Cameras on Track, a drone and maybe the in vehicle cameras.

There is our first test this year, which started bad, but got better in the end.
https://youtube.com/playlist?list=PLJ-leheBA7Kg&si=0uBTGvQExG0eVx-8

[BrandyWine]

Are you building this for a specific car/team, or is this a service to all car/teams?

You said no 5GLTE, so how then do you reach utoob?

It's a racetrack, are there any local stipulations where you can increase wifi Tx power? How does Indy or NASCAR do in-car video? Wifi is cheap, but perhaps comes with it's own problems due to Tx power limits.

From my view, all the camera feeds should land in a 2x2 video mixer, one that allows you to show (choose) 1, 2, 3, or 4 video feeds at once, basically allowing you to have 4 video feeds in one HD screen (car, drone, cam1, pit), from there it connects (outputs hdmi) to a live-stream gateway that encodes direct to utoob live, same encoder gateways can live stream to multiple platforms at the same time. The encoders do usually support 5g-wifi and ethernet-wired, so some choices there.

As for multi-wan, sure, OPNsense can do that via policy-based routing. You default route based on source subnet. 10.10.10.0/24 DFG WAN1, 10.10.11.0/24 DFG WAN2, 10.10.12.0/24 DFG WAN3, etc. This also implies that you have three distinct connections or ISP's for internet access.

To do that, I would use a device that has at least 6 gig ifaces, 3 for WAN and 3 for LAN, then just match them, like LAN1-WAN1, LAN2-WAN2, LAN3-WAN3.

Having the fw in there allows you to control the data flows. Duly noted the single point of failures.