Messages

Lab is back up and I'm getting things running again... Which one is likely to be fastest? That said, I have 200mbps down on a good day, more likely 100mbps down and maybe 50mbps up. Running this on my mifi device because it made life more simple.

I may try many of these to see the speed difference, I'm now running on an HP T740 with 16gb of ram (AMD V1756b processor), also on a quad port intel x710 with gigabit adapters installed. I may change to 10gb for LAN-LAN traffic, but clearly gigabit is plenty fast for my WAN connection. Harvester and Truenas Scale are up and ready to do some work.

But first things first, I need to get Netbird set back up on this "new" firewall so I can get back in from home.

Regarding the price, I recently paid a bit over $900usd for Protectli i5 with 16GB of ram, some smallish SSD, and 6 ports of i226. I would have purchased an OPNsense hardware, but they are really hard to get in the USA, and the recent prices with tariffs made them impossible. So the Mono price at $600 might not be that far out of line, but for my lab, it was more than I was willing to go right now. I have many HP T740 and cards, and an HP T620+ with 4 port card that still seems to mostly work. Lab is swapped to a T740 with 16GB, 120GB ssd, and quad port i710 for 1/10g connections. Total for this was around $300 when I bought all the parts, but most of that was before the rampocalypse.

I went to an AC charger to keep the mifi device on power. I'll have to look at the data blocker.

I haven't had time to look through logs to see why the USB into the firewall was causing an issue, I'm guessing it was detecting this and considering it wan, but not really getting it configured. The mifi has the ability to use either USB or ethernet to your firewall, so I'm guessing that even though I turned off the USB path, it was still allowing enough data to fool the firewall.

It is interesting, especially VPP support. Routing across local LANs at 10g with less overhead would be nice, something I may start doing in my lab just to see what the results might be. My current lab OPN device has a quad port x710 card installed, so I could get 3 LAN all going at 10g if I wanted to test it, WAN is still very limited.

There was some discussion on the Lawrence Systems forums about pf and the amount of code the submit back to BSD. A couple mentioned that OPN also contributes code back, but that there were politics that got in the way.

Sad that they are still being petty.

In order to run the Vulkan version of LoacalAI, I needed to run this in a container (docker), but there is a bunch of messing around to feed the GPU through to the container, same with persistent storage. No problem sharing the display with the container, that's the only way I've tested it so far (going to 127.0.0.1:8080)

I'm running mine on an HP T740 with the internal gpu. Speeds are pretty slow, but it's working. With 32GB of ram, I'm able to load a few different 8GB models (not at the same time). It's working, I may trade RAM from another T740 and increase this up to 64GB even though it might slow things down moving to another bigger model. Just kind of playing right now while I decide what to do, and how much money to spend. Wishing I had bought a used AMD V620 when they were still available at around $300usd. Would need to mess around with PCIe extension, power supply, and cooling for the card, but might have been worth it.

Coming back around to this because I had a problem that I'm now trying to understand.

OPNsense works fine when the ethernet connection is plugged it, doing an update right now.

But if I connect the USB to keep the battery charged, the entire firewall stops all traffic. The local console is working, but DHCP server on management port doesn't work. Even if I set the IP manually on a connected computer, there is no connection and can't ping anything. I have the USB connection turned off (charge only) in the menu of the mifi device, but apparently that's not good enough. I've used the mifi in the car and it works fine while charging, so not a limitation there.

I'll have to find a charge only cable, or find a spot for a USB power supply in my rack.

Any other ideas where I should look?

What it also is very clever at, is making suggestions at how I can add and improve things. It wasn't my idea to add all of the monitoring

I only have experience of the free online LLMs and very recently running local models with LM studio. So far, I have only used it to generate (hesitate to use the word create!) standalone or loosely coupled shell or python scripts.

I'm getting started with small models running on severely underpowered computers, but it's working, it's local, and I can dig away at things for as many hours as I care without the thought of tokens running out. The light/basic tasks I'm asking it to do are working and results seem to be correct. Right now I'm still mostly asking questions that I already know an answer for, checking its work to make sure I can generally trust it. I'm using LocalAI in Vulkan mode which seems to be the highest performance I can get out of the little machine I'm using.

I got a response from Protectli:

#1 my VP4650 shouldn't have this problem.

#2 they offered another document regarding ASPM for the older devices:
https://kb.protectli.com/wp-content/uploads/sites/9/2026/02/TSB-2025-001_VP2440-ASPM-Network-Interface-Performance-Issue-v2_0.pdf

Hopefully they watch this thread to see the outcome and distill it back into another KB document.

I think it is more simple for the Intel cards, I think you can just set all interface options and it works.

But the mod you mention I think is the SMBus or I2c bus, I've seen that with a few other cards in main boards that don't fully support those protocols.

Uggg... I just received a brand new VP4650 with 6x i226 ports to replace my old recycled OPNsense server. I'm also on Business so fixes might be a while.

I guess I wait and maybe burn it in on my lab?

I'll be watching this. And before I bought this the sales person spoke to Protectli, and I spoke to Protectli about using this with OPNsense, the only thing they said was that I probably want to run Coreboot, but can switch back and forth at any time.

Considering the difficulty of people in the USA for getting Deciso based hardware, Protectli is about the best choice. And they were on government contract which helped me buy it for work. Seems like there might be a spot for more of a partnership between Protectli and Deciso/OPNsense.

[edit] I sent an email to the sales contact I made a few months ago, linked this thread. Maybe we can get Protecli on this to help out. The OPNsense hardware wasn't really an option for me here in the USA, which is why I ended up with a Protectli device.

There are ways to unlock the cards, Serve The Home has a few in the forums and generally all you need are the Intel provided tools. That's for another day, I really need to teach myself about this process because you never know what you are going to get when buying used equipment like so many of us have to do for our labs.

We mostly do not need any of the SASE features

I'll disagree, lab is where you want to fool with these features the most, then decide what and how much to put into production.

I'll have to look into the SASE starter for my lab, just getting a new to me hardware set up and would really like to look into options for production going forward.

The x710 has been good with most DAC and mulitmode optical, but when you need 1g copper, you need 1g copper. Addon was the brand that works, not even all of the FS were working. I even have some "real" Cisco GLC-T, and newer "off brand" that work in my Mikrotik 10g switches that allows going down to 100m for one device that still only has 100m.

I did update the firmware, but I'm having problems looking deeper on BSD, no ethtool in the OPNsense OS, and no indications of problems in the INTEL tools under Win10. I'll have to boot up a linux live and see if I can make heads or tails out of ethtool and see why these cards are so picky.

These Addon SFP were from ebay, I need to look for a few more. I think I paid $2-$3 each for them (shipped). And they were bought to try and get that 100m device connected (Clockwork Pi uConsole with ethernet card).

IPPT (IP passthrough) seems to work fine, and you can keep the wifi going as normal which has been handy with grandkids in the car.

Since this is still CGNAT, the only thing this really gives is one less NAT layer, but it worked immediately with no messing around on the ethernet connection. Haven't tried with the USB connection yet, and probably wont bother since this is working.

Now if I can ever get my lab put back together and get the new firewall in place, a new to me HP T740 with very picky x710-da4 card. Had a difficult time finding SFP gigabit modules that would work, I only have about 6 different brands and compatibility firmwares.