The joys of Google AI

[Greg_E]

In order to run the Vulkan version of LoacalAI, I needed to run this in a container (docker), but there is a bunch of messing around to feed the GPU through to the container, same with persistent storage. No problem sharing the display with the container, that's the only way I've tested it so far (going to 127.0.0.1:8080)

I'm running mine on an HP T740 with the internal gpu. Speeds are pretty slow, but it's working. With 32GB of ram, I'm able to load a few different 8GB models (not at the same time). It's working, I may trade RAM from another T740 and increase this up to 64GB even though it might slow things down moving to another bigger model. Just kind of playing right now while I decide what to do, and how much money to spend. Wishing I had bought a used AMD V620 when they were still available at around $300usd. Would need to mess around with PCIe extension, power supply, and cooling for the card, but might have been worth it.