Colibri: Run Huge MoE Models Locally on Your PC

You have an everyday laptop or PC: 16 GB or 32 GB of RAM, an NVMe drive with a few dozen gigabytes free. You want to try a frontier MoE model such as GLM-5.2 or Kimi K3 on that machine — data stays local, no API fees, no dependence on the network. Then you open the model page and stop: the model weighs hundreds of gigabytes, your graphics card has only a few gigabytes of VRAM, or you have no discrete GPU at all. The familiar approach — loading the whole model into VRAM — is closed from the start.

But if each token only activates a small part of an MoE model, why keep the entire model in VRAM? Colibri answers that question: keep only the part in use in RAM or VRAM, and stream the rest from disk when needed. A model with hundreds of billions of parameters can thus run on the machine you already own — provided you accept speeds much slower than running resident on a dedicated GPU.

Note: I have not installed or measured Colibri myself; every speed figure in this article is the project’s own measurement, not a benchmark from this blog.

What is Colibri, and what problem does it solve?

Colibri is a pure-C inference engine that runs very large open MoE models on the machine you already own, with no GPU required — “tiny engine, immense model”, in the project’s tagline. As of 2026-11-03, JustVugg/colibri uses the Apache-2.0 license, version v2.1.0 (2026-10-31), about 46,968 stars, and remains actively maintained.

The problem lies in the MoE architecture. A dense model uses nearly all of its parameters for every token, so the memory needed is roughly the size of the model. An MoE model is different: each token passes through only a small set of experts at each layer, so the parameters actually involved in one token are far fewer than the total. Colibri turns “not enough VRAM” into “is there a fast enough drive and enough RAM for a cache?”

It fits sysadmins, DevOps/SRE engineers, backend/full-stack developers, homelab self-hosters, and coding-agent users who want an offline LLM without renting a datacenter GPU. It does not fit production workloads needing high speed or many concurrent users, or machines with only a slow HDD/SSD and little free space.

Colibri architecture: streaming MoE experts from NVMe

flowchart LR
    A["Prompt / token in"] --> B["Per-layer router:\nselect experts for this token"]
    B --> C{"Which tier holds\nthe expert?"}
    C --> D["VRAM — hot set:\npinned hot experts"]
    C --> E["RAM — warm cache:\nper-layer LRU"]
    C --> F["NVMe — cold store:\nfull model"]
    F -. "Cache miss: load to RAM / VRAM" .-> E
    E -. "Frequently used experts" .-> D
    D --> G["Layer computation\nand token output"]
    E --> G
    F --> G
    B -. "Prefetch: preload experts\nfor the next layer" .-> F
    G --> H["Learn: update\n.coli_usage and hot set"]
    H -.-> D

Each token passes through every layer, the router selects experts, the system fetches each expert from the fastest tier that holds it and prefetches the next layer’s experts. Three tiers: VRAM (pinned hot set), RAM (per-layer LRU cache), NVMe (the full model, one pread for the three matrices of each expert).

The key point: placement only decides speed, not which expert the router picks or the precision of the weights. A cache miss slows things down; it does not make the result wrong. You can therefore measure and improve Colibri like a cache — hit rate per layer, warmth of the hot set — rather than as a GPU-buying problem.

The per-token pipeline is: route (select experts), union (merge demand), place (put experts in the right tier), overlap (load experts while computing), learn (learn from conversations, write to .coli_usage, pin the hot set for next time).

The engine is pure C, one file per model family, with CPU, Vulkan, and CUDA backends, plus GGUF files for language models. DIRECT=1 often helps a lot on fast NVMe but can be worse on other drives, so measure on your own drive; COLI_MODEL_MIRROR lets you add a second drive.

Limits: a cold run is much slower than a warm run; each model takes tens of gigabytes of disk; Qwen-Image-2.1 is non-commercial only.

Which models run on which machines?

Verified scope: GLM-5.2/5.3, Kimi K3, DeepSeek V4 Flash/V4.1 Flash, Qwen3.6, Qwen3-Coder, OLMoE, Qwen-Image-2.1, Laya, GLiNER2.5-Decide. Minimum requirements: 8 GB RAM (16 GB or more is more comfortable), 22 GB free for the smallest model, Internet needed while downloading. GPU optional.

The lower RAM figure is the runnable level, the higher one the comfortable level. For GLM-5.2 744B, the download is much larger, but the principle is unchanged. Instead of guessing, run ./start-here.sh --list to see which models fit your machine.

Installing Colibri and running a local MoE model

Verified version: Colibri v2.1.0 (2026-10-31), checked 2026-11-03. Before installing, check RAM, free NVMe space, and Internet access.

Installing on Linux

The commands below were cross-checked against the project’s official documentation:

sudo apt install git python3 build-essential
git clone https://github.com/JustVugg/colibri
cd colibri
./start-here.sh

Setup inspects the machine, suggests a fitting model, builds the engine if a compiler is present (otherwise downloads a prebuilt build), downloads the model with progress and resume support, then starts Colibri and opens the dashboard. Useful options: --list, --model ID, --dir DIR, --backend vulkan|cuda|cpu, --no-gpu, --reconfigure.

Installing on macOS and Windows

On macOS: install the Xcode command line tools and required packages via Homebrew, then clone the repo and run ./start-here.sh. On Windows: download the ZIP from the repo page and run START-HERE.bat (installing Python is suggested if missing). For WSL: avoid placing models on /mnt/c, where disk reads are slow.

xcode-select --install
brew install libomp git python

Verifying a successful installation

After setup finishes, check with c/coli status and stop with c/coli stop (Windows: c\coli.cmd status/stop). Expected result: dashboard at http://127.0.0.1:8000/, OpenAI base URL at http://127.0.0.1:8000/v1, Anthropic base URL at http://127.0.0.1:8000. Try it with ./coli chat. The project’s example: GLM-5.2 744B MoE int4 streaming on CPU “ready in 32s, resident 9.9 GB” — the project’s example, not a result measured by this blog.

From here, point a coding agent or OpenAI-compatible application at that base URL for coding, review, or summarization. If you already self-host a gateway layer, Colibri is the inference engine behind it. Read more: self-host New API as an AI gateway.

Technical assessment: pros, cons, security, and performance

The clearest advantage is running MoE models with hundreds of billions of parameters on a machine you already own, with no datacenter GPU, no API fees, and data staying on the machine. Installation is one step with start-here.sh, with resume if a download is interrupted. The OpenAI- and Anthropic-compatible APIs on localhost plug straight into existing tools. The source is open under Apache-2.0, with GGUF, CPU/Vulkan/CUDA support.

The downsides must be stated fully. Speed is much lower than running a resident model on a dedicated GPU, and the experience depends heavily on the drive and cache state. Each model download costs tens of gigabytes of disk. The project is young and fast-moving — v2.1.0 only arrived in late October 2026 — so model names and options may change; recheck the current documentation.

All performance figures below are the project’s measurements on its own hardware. For GLM-5.2, decode reaches 5.8–6.8 tok/s on 6x RTX 5090, 1.83 tok/s warm on CPU-only 128 GB, 1.07 tok/s on an RTX 5070 Ti laptop, and a cold floor of only 0.05–0.1 tok/s. For Qwen3.6-35B-A3B: 6.0 tok/s on CPU, 9.9 tok/s on Vulkan iGPU, and 30.0 tok/s on CUDA. Do not use these numbers to infer the speed on your machine.

In my view, the upgrade order is the most important insight. With streaming MoE, actual VRAM/RAM only needs to cover the active experts per token plus the KV cache; the total parameter count becomes a question of NVMe capacity and random-read bandwidth. So prioritize a faster NVMe, measure DIRECT=1 in practice, then a larger RAM cache, before a more expensive CPU or GPU. When decode is disk-bound, adding CPU cores does not solve pread wait time, and heavily scattered routing causes cache thrashing, dragging speed back toward the cold floor.

On security and privacy, the real benefit is that model and prompts run locally, with nothing sent to an external API when you use the correct localhost endpoint. But the default addresses bind 127.0.0.1, and the documentation does not describe an authentication mechanism for the local endpoint. So never expose port 8000 to the LAN or the Internet; if remote access is needed, put the endpoint behind a reverse proxy with authentication — never open the port bare.

Comparison with llama.cpp, Ollama, and GPU inference servers

Criteria: hardware, large-MoE handling, speed, ease of installation, API, use case — at the level of principle, with no invented head-to-head benchmarks.

• Colibri: streams MoE experts across VRAM/RAM/NVMe tiers, with prefetch and a learned hot set; choose it to run a very large MoE model on the machine you already own, GPU optional.
• llama.cpp / Ollama: general inference with GGUF, a broad ecosystem, simple to use; choose them for a general GGUF ecosystem.
• vLLM / SGLang: throughput and concurrent serving on dedicated GPUs; choose them for GPU production.

What is Colibri actually used for?

For a homelab, Colibri gives you a private AI lab at home: run a local assistant, try large models without API fees, place models on NVMe and separate the model folder with --dir. For coding agents, point your existing tool at http://127.0.0.1:8000/v1 for coding, review, and summarization. For a business, it is a way to validate an MoE model on machines you already own before buying GPUs or renting cloud capacity.

Common errors, backup, and rollback

Interrupted download: run setup again to resume; do not delete the model folder mid-download. Missing GPU packages: install the packages from the command the documentation prints, then run setup again. Out of disk: use --dir/--model-dir to point at a drive with free space, preferably NVMe. Slow running: check the drive type first, try DIRECT=1, measure again, and determine whether the state is cold or warm.

Rollback: stop with c/coli stop and delete the model folder in ~/colibri-models or the --dir you chose; no need to uninstall the engine. On a slow drive, fall back to a cloud API or a smaller model such as Qwen3-Coder-30B-A3B.

Conclusion: who fits, who does not, and the next step

Fits: people with tens of gigabytes free on NVMe who accept streaming speed in exchange for privacy and no API fees. Does not fit: people who need fast answers for production, serve many users, only have a slow HDD/SSD, or do not want to download tens of gigabytes.

The lowest-risk next step: run ./start-here.sh --list, pick the smallest fitting model (for example Qwen3-Coder-30B-A3B), measure warm speed after a few conversations, and only then decide whether to download a larger model.

Lessons learned

1. Model size is no longer an absolute barrier: with MoE, the right question is which experts are hot.
2. The drive is the main performance resource: a fast NVMe and a measured DIRECT=1 are worth more than blindly upgrading the CPU.
3. The first run does not represent steady speed: warm speed differs sharply from cold speed.
4. API compatibility is operational value: the localhost endpoint lets Colibri plug into existing tools.
5. Do not expose a local endpoint without an authentication layer in front of it.

Verification scope

Check date: 2026-11-03. Main sources: the README and GitHub API of JustVugg/colibri, documentation at justvugg.github.io/colibri, version v2.1.0 released 2026-10-31. Verified: minimum requirements, install commands, endpoints, CLI options, model list, and the RAM/download table. Defaults: GPU is optional, setup inspects the machine and suggests a model; DIRECT=1 and COLI_MODEL_MIRROR are opt-in depending on the drive; GGUF is a supplementary path for language models. Speed figures are the project’s measurements, not this blog’s benchmarks. This article did not install, run, or measure Colibri itself. Not verified: the star count at publication time, a Vietnamese-language interface, and speed on the reader’s machine.

References

• Colibri on GitHub: https://github.com/JustVugg/colibri
• Colibri documentation: https://justvugg.github.io/colibri