Nov Deploying Open-Source Models Locally

With 64GB of RAM, you can run DeepSeek V4.1 Flash using llama.cpp with Unsloth's GGUF quantized version, lowering the barrier to consumer devices.

Large MoE models can run on gaming GPUs using FreeToken. The expert cache is stored in VRAM, and the rest of the computation is split between CPU and GPU according to bandwidth.

A 24GB GPU can handle long texts with a modified version of Qwen3.8-27B that uses pure linear attention. At 4-bit it's about 17GB, and VRAM barely grows with context length.

On Mac, prioritize MoE models: Qwen3.6-35B-A3B runs at 63 to 69 tokens/s, while the dense 27B manages only 11 to 12.

When buying Mac memory, decide on your model first. The 16GB version is only enough for around 12B-class models. For 30B-class models, 24GB is a recommended starting point, since memory is soldered and can't be upgraded later.

If you're unsure about model size and quantization level, use llmfit to first detect VRAM and memory bandwidth, then get recommendations for runnable models at 4-bit or 8-bit.

The open-source single-image-to-3D project Idol takes about 57 seconds on a 5060 Ti, with peak VRAM just over 7GB. Its output is Gaussian splatting rather than editable meshes.

For local 3D work on Mac, you can choose Hunyuan3D 2.1, or TRELLIS.2, which has a community Mac version. TRELLIS.2 is under the MIT license and can be used commercially.