Inkling 2 Small Opens Apache 2.0 Weights: A 594B MoE You Can Download — But Who Can Run It?

The real problem: you want a strong open model, but strong models are closed

Last week, a coding agent on a mid-sized open model helped me split the authentication module out of a monolith — until it forgot the agreed interface, called a tool with wrong parameters, and broke two tests.

Strong coding-agent models are usually closed: your code leaves your network, and fine-tuning is off the table. Open models that fit a personal PC are weak exactly where it matters — long reasoning, tool use, and context big enough for a codebase.

Inkling 2 Small from TML lands in that gap: on 12 November 2026, TML released the weights on Hugging Face under Apache 2.0 — a 594B MoE, the highest-scoring open-weights model outside China on the AA Index, with a 1M-token context. But open weights do not mean everyone can run it.

What is Inkling 2 Small?

Launched 5 November 2026 as hosted-only through Tinker — TML’s own API — with weights promised “next week, after safety testing and red-teaming”. On 12 November 2026 (~20:50 UTC), they landed on Hugging Face in thinkingmachines/Inkling-2-Small, under Apache 2.0: 25 safetensors files, main checkpoint NVFP4, plus a separate MXFP8 repo. Apache 2.0 allows self-hosting, fine-tuning, and commercial use with no license fees — the barrier left is hardware, not licensing.

Benchmarks

Who should use it: teams with an H200/B300 cluster or inference providers. Who should not: homelabs, chatbot-only users, or offline use — BotPsy 0.8B is the better choice.

Architecture: 594B MoE, 17B active, 1M-token context

A 48-layer decoder-only MoE: 594B total, 17B active per token (router picks 8 of 384 experts), hybrid local + global attention; text and images in via a vision encoder, text out, no audio. Inference cost scales with the 17B active, but storage and loading scale with the full 594B — every inference streams all weights through VRAM, so memory, not FLOPs, is the bottleneck.

NVFP4 vs MXFP8

Architecture diagram: inference flow and deployment paths

flowchart TD
    A["User / Coding agent<br/>Prompt + tool definitions"] --> B["API Gateway / vLLM<br/>OpenAI-compatible endpoint"]
    B --> C["Router MoE<br/>48-layer decoder"]
    C --> D["Hybrid attention<br/>local + global"]
    C --> E["Vision encoder<br/>image → embedding"]
    C --> F["17B active<br/>8/384 experts per token"]
    F --> G["DSpark speculative decoding<br/>7 tokens, acceptance ~3.5"]
    G --> H["Reasoning parser inkling"]
    H --> I["Tool-call parser inkling<br/>+ auto-tool-choice"]
    I -->|Needs tool| J["Tool execution<br/>shell / API / MCP"]
    J --> C
    I -->|No tool needed| K["Text output"]
    B --> L["A. Self-host vLLM / TokenSpeed<br/>NVFP4 ≥500GB VRAM, TP8 = 1M ctx"]
    B --> M["B. API — Tinker $0.30/$1.00 per 1M tok<br/>reasoning_effort"]
    B --> N["C. Wait for community quants<br/>GGUF / Unsloth — no official build recorded (13 Nov)"]

Limits to know first

• 500GB minimum VRAM (NVFP4) is a hard barrier — no configuration trick reduces it; wait for community quants, of which no verified official GGUF build existed as of 13 November.
• 1M-token context is not automatic: on the GB200-verified vLLM configuration, TP4 tops out at 262K tokens; only TP8 reaches 1M.
• Fine-tuning is limited: LoRA works on attention projections and routed experts, but NVFP4/MXFP8 checkpoints allow attention-only adapters.

Self-hosting: GPU requirements, vLLM / TokenSpeed

To be blunt: almost no homelab reader qualifies — datacenter GPU-cluster territory. For scale: 100M input + 50M output tokens via Tinker cost $80/month.

vLLM — day-0 support

vLLM supports Inkling 2 Small from day zero — but until support lands in an official release, use the dedicated image vllm/vllm-openai:inkling-2-small:

pip install -U "huggingface_hub[cli]"
huggingface-cli download thinkingmachines/Inkling-2-Small

Serve command, from the vLLM blog (12 Nov 2026):

vllm serve thinkingmachines/Inkling-2-Small \
  --tensor-parallel-size 8 \
  --reasoning-parser inkling --tool-call-parser inkling --enable-auto-tool-choice \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'

--reasoning-parser inkling and --tool-call-parser inkling are mandatory — without them, tool calls fail or reasoning leaks into the output.

TokenSpeed on 4x B300

TRITON_USE_SLP_VECTORIZER=1 ts serve thinkingmachines/Inkling-2-Small \
  --tp 4 --ep-size 4 --mm-encoder-tp-mode data \
  --max-model-len 1048552 --sampling-backend triton \
  --kv-cache-dtype mxfp8 --speculative-algorithm DSPARK

First startup autotunes kernels ~10 minutes on 4x B300.

Verify the deployment

Skip GET /health as a readiness check: vLLM returns 200 as soon as the server is up, before weights finish loading, and 503 only when the engine dies. Use GET /v1/models — 503 until the model is ready — or a test chat completion.

curl -s http://localhost:8000/v1/models

Then send a test request to /v1/chat/completions with model thinkingmachines/Inkling-2-Small, requesting reasoning and a sample tool call.

Common errors

• OOM / insufficient VRAM, or context cut below 1M: MXFP8 needs 800GB — switch to NVFP4; check tensor-parallel size and KV cache dtype; you may be on TP4 (262K limit) instead of TP8, or max-model-len unset.
• Reasoning/tool-call in the wrong format: missing --reasoning-parser inkling or --tool-call-parser inkling.

SGLang, Unsloth, and Hugging Face are also listed — unverified.

If you cannot self-host: the API

Tinker API

Tinker is TML’s official API; install from PyPI:

uv pip install tinker-cookbook

Set model_name="thinkingmachines/Inkling-2-Small" and TINKER_API_KEY (from the Tinker console) via environment variables, never hardcoded; reasoning_effort is adjustable.

The 1M-token context exists only when self-hosting: on Tinker, the model serves 64K standard or 256K extended context (model ID suffix :peft:262144) — do not design a RAG workflow around 1M tokens on the API.

Tinker pricing: $0.30/$1.00 per 1M input/output tokens; Together charges the same. Other providers (NVIDIA, Fireworks, Baseten, Nebius, Modal, Cloudflare) are listed per the model card, prices unverified.

Output tokens cost ~3.3x input tokens, and for a coding agent most cost sits in reasoning tokens — billed at output rates — so set reasoning_effort per task difficulty: low for routine tasks, high only for hard ones.

Evaluation: strengths, weaknesses, risks

Strengths: genuinely open weights under Apache 2.0 — self-host, fine-tune within LoRA limits, use commercially with no license barrier; strong on coding agents by vendor benchmarks (SWE-bench Pro v2 Hard 86.3%, BrowseComp 90.2% with compaction); complete day-0 ecosystem. Weaknesses: 500GB minimum VRAM — “open” on license, closed on infrastructure for most users; MXFP8 costs 300GB more VRAM for 1–2 eval points; attention-only LoRA; no audio; vendor-reported benchmarks; safety/red-teaming details unpublished.

Security when self-hosting

Apache 2.0 weights are not safe by default: self-hosting means you own guardrails, filtering, rate limiting, and audit logs. The model card reports safety evaluation and external red-teaming with no “material uplift” beyond the current open-weight ecosystem — not “absolutely safe”.

Three concrete risks: an unauthenticated endpoint will be abused to mine tokens — hide it behind a reverse proxy with an API key and rate limits; prompt injection via tool calls — malicious instructions can trigger unintended tools, so restrict tool permissions, whitelist shell commands, and log every call; supply chain — download weights only from the official Hugging Face repo, and beware mirrors or unverified quants.

Rollback: keep the old checkpoint and vLLM image when upgrading; deploy API first, self-host later. Verified performance: ~630 tokens/s per user at TP4 on GB200 with DSpark (acceptance 3.5), per vLLM.

Comparison with alternatives

Inkling 2 Small wins on “strong + open” and loses on “anyone can run it” — Colibri is the closest comparison: both are open MoEs, one runs on a personal machine, the other needs half a terabyte of VRAM.

Real use cases

Enterprise / SRE: self-host inside a VPC so code never leaves the network, with a 1M-token context swallowing the whole repo — the most sensible case with a GPU cluster or rental budget.

AI builders: install the Tinker cookbook, set reasoning_effort per task difficulty, and measure the bill on real tasks; self-host when monthly API cost exceeds cluster rental.

DevOps: monitor VRAM, tokens/s, acceptance, and tool-call errors.

Conclusion

A good fit: teams with a datacenter GPU cluster, inference providers, and AI builders who need a strong open model and will pay $0.30/$1.00 per 1M tokens. Not a good fit: homelabs with gaming GPUs, offline PC use, deep fine-tuning, audio input.

Next steps: (1) try the Tinker API with low reasoning_effort to measure real cost on your own tasks; (2) with a GPU cluster, try the vLLM TP8 recipe; (3) without enough GPUs, follow community quants (Unsloth, llama.cpp) instead of forcing the full weights in.

Lessons learned

1. Open weights do not mean everyone can run it: Apache 2.0 is open in legal terms, but 500GB minimum VRAM is the real barrier — the immediate beneficiaries are inference providers with B300/H200 clusters.
2. With an MoE, the bottleneck is memory, not FLOPs: every token still streams the 594B weights — that is why TML recommends NVFP4 and vLLM uses DSpark.
3. “1M context” comes with conditions: TP4 gives only 262K tokens; only TP8 reaches 1M — check the configuration before designing RAG around 1M tokens.
4. The most effective cost knob via API is reasoning_effort: output tokens cost 3.3x input tokens, and reasoning tokens are billed at output rates — set effort per task difficulty.
5. Do not trust benchmark numbers before testing them yourself: the launch post says AA Index 31, the model card says 30.6 — all numbers are vendor-reported; test on real tasks via the API before investing in a GPU cluster.

Verification scope

Checked: 13 November 2026. Cross-checked against the Hugging Face model card, the official vLLM blog and recipes (12 Nov 2026), TokenSpeed recipes, the tinker-cookbook PyPI page, and Tinker and Together pricing. Readiness uses GET /v1/models, not /health — which returns 200 before the model finishes loading. Benchmarks are vendor-reported by TML, with no independent measurement; 630 tokens/s and 3.5 acceptance length are vendor-published GB200/TP4 figures. Tinker pricing ($0.30/$1.00) and the 64K/256K context limits were verified from Tinker’s pricing page; other providers’ prices were not. Self-hosting was not tested in practice. Detailed safety/red-teaming results remain unpublished.

References

• Model card: https://huggingface.co/thinkingmachines/Inkling-2-Small
• vLLM blog: https://vllm.ai/blog/2026-11-12-inkling2-small
• vLLM Recipes: https://recipes.vllm.ai/thinkingmachines/Inkling-2-Small
• TokenSpeed: https://lightseek.org/tokenspeed/recipes/models
• PyPI tinker-cookbook: https://pypi.org/project/tinker-cookbook/
• Tinker pricing: https://thinkingmachines.ai/tinker/pricing/
• Together pricing: https://together.ai/pricing