In short: Thinking Machines released the weights of Inkling 2 Small on Hugging Face on November 12, 2026 under Apache 2.0. It is a 594B-parameter Mixture-of-Experts model with 17B active parameters, so "running it locally" means a multi-GPU server with at least 500GB of combined VRAM, not a laptop. For most people the hosted APIs at $0.30 per million input tokens and $1.00 per million output tokens are the practical route; self-hosting only pays off at very high, steady volume or when the data cannot leave your own hardware.
What Inkling 2 Small is, in numbers
Inkling 2 Small is the first model of Thinking Machines' second Inkling generation. The facts that decide which route fits, all checked on the official announcement and Hugging Face model card this morning:
- Size: 594B total parameters, 17B active per token. Each token is routed to 8 of 384 experts across 48 decoder layers.
- Inputs and outputs: text and image in, text out. There is no audio input, unlike the first-generation Inkling models.
- Context: up to 1M tokens (1,048,576) according to the model card and the vLLM recipe.
- Reasoning effort: variable. Thinking Machines ran its own evaluations at effort 0.99 ("xhigh").
- License: Apache 2.0, shown on the Hugging Face model page.
- Checkpoints: the main
thinkingmachines/Inkling-2-Smallrepository in NVFP4 (the precision the model was trained in), andthinkingmachines/Inkling-2-Small-MXFP8. The model card recommends NVFP4 for all use cases and calls the 1–2 point gain of MXFP8 on evals not meaningful in practice.
The announcement was first published on November 5 with Tinker-only access; the November 12 update added the open weights plus integrations with SGLang, vLLM, TokenSpeed and Unsloth, and a list of API partners.
Route 1: Download the weights and self-host
Check the hardware first
The model card is blunt about it. The NVFP4 checkpoint needs a GPU cluster with at least 500GB of aggregated VRAM, for example 2x NVIDIA B300 or 4x NVIDIA H200. The MXFP8 checkpoint needs at least 800GB, for example 4x B300 or 8x H200. I queried the Hugging Face file API to see what you actually download: the NVFP4 repository holds 393.5 GB of safetensors in 25 shards, and the MXFP8 repository holds 622.8 GB in 49 shards. Budget disk space and download time accordingly, and keep both on fast local storage rather than a network mount.
The SGLang cookbook adds useful detail: at tensor parallel 4 the NVFP4 weights take about 94 GB per GPU and the MXFP8 weights about 151 GB per GPU, which is why 192GB cards such as B200 run the MXFP8 version across a full 8-GPU node.
Option A: vLLM
The official vLLM recipe (updated November 12) says to use the dedicated vllm/vllm-openai:inkling-2-small Docker image until support lands in a regular vLLM release. Its default configuration is NVFP4 on 4x B200 with tensor parallel 4:
docker run --gpus all --privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:inkling-2-small thinkingmachines/Inkling-2-Small \
--tensor-parallel-size 4 \
--max-model-len 262144 \
--max-num-seqs 256 \
--reasoning-parser inkling \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
Notes from the recipe that matter in practice: one GB200 tray at TP4 was verified for contexts up to 262,144 tokens, and you need two trays (TP8) for the full 1M window. The checkpoint ships with a bundled DSpark speculative-decoding draft. Tool calling uses the inkling tool-call parser, which the recipe says is unit-tested but not yet benchmarked end to end. Once the server is up, it speaks the OpenAI-compatible chat API on port 8000.
Option B: SGLang
The SGLang cookbook has verified launch commands for GB300, GB200, B300, B200, H200, AMD MI350X/MI355X, Vera Rubin and TPU7x. The Blackwell recipes run TP4/EP4; H200 and AMD run TP8/EP8. A trimmed version of the GB300 NVFP4 command:
sglang serve \
--model-path thinkingmachines/Inkling-2-Small \
--tp 4 --ep 4 \
--moe-a2a-backend megamoe \
--attention-backend fa4 \
--enable-multimodal \
--reasoning-parser inkling \
--tool-call-parser inkling2 \
--host 0.0.0.0 --port 30000
Two gotchas from the cookbook: keep EP equal to TP, because the NVFP4 experts cannot be tensor-sharded, and the MegaMoE backend needs a DeepGEMM build with Inkling 2 Small support, otherwise the server fails at load. If you do not have that build, pick the routed-experts backend instead. On AMD, only gfx950 GPUs (MI350X/MI355X) are validated; other ROCm GPUs refuse at launch.
Option C: Unsloth and smaller quantizations
The announcement and model card both list Unsloth as an integration, and Unsloth has published low-bit GGUF versions of the first-generation Inkling models before. But when I checked this run, Unsloth's Inkling docs page still covered only Inkling and Inkling-Small, and I could not find a public Inkling 2 Small GGUF repository under Unsloth's Hugging Face organization. If you were hoping for a Mac Studio-sized 1- or 2-bit build, treat that as "not yet" and check back, rather than assuming the first-generation memory table applies.
Route 2: Use a hosted API
The announcement lists seven API partners: NVIDIA, Fireworks, Together, Baseten, Nebius, Modal and Cloudflare. I could confirm a live model page with prices for four of them, plus Thinking Machines' own Tinker. The Baseten and NVIDIA model URLs linked from the announcement returned "not found" pages when I checked, and I could not find Inkling 2 Small on Fireworks' public model or pricing pages, so they are left out of the table below rather than guessed.
- Together AI: model
thinkingmachines/Inkling-2-Small, serverless and dedicated, OpenAI-style chat completions endpoint. - Modal: a shared, OpenAI-compatible endpoint billed per token; Modal's page shows the one-line
modal endpoint create --shared --model thinkingmachines/inkling-2-smallsetup. - Nebius Token Factory: a public shared endpoint serving an FP4 quantization, with tool calling, reasoning and Responses API support. Nebius itself says the shared endpoint is meant for tests, not production, and that its region may change.
- Cloudflare Workers AI: model ID
@cf/thinkingmachines/inkling-2-small, with function calling, vision, reasoning levels from none to max, and zero data retention. It is not available on the Workers Free plan; you need Workers Paid or prepaid AI Gateway credits. - Tinker: serverless inference is in beta, and Thinking Machines runs a waitlist for production use. Tinker is also where you fine-tune the model ($1.50 per million training tokens at 64K context, $3.00 at 256K).
Comparison: which hosted route costs what
| Route | Input / cached / output (per 1M tokens) | Free tier | What stands out |
|---|---|---|---|
| Together AI | $0.30 / $0.03 / $1.00 | None listed for this model | Serverless and dedicated; 1M context |
| Modal shared endpoint | $0.30 / $0.03 / $1.00 | None listed for this model | Pay per token, OpenAI-compatible; option to deploy your own GPUs on the same platform |
| Nebius Token Factory | $0.30 / $0.06 / $1.00 | None listed for this model | FP4 quantization, ~130 tok/s shown; shared endpoint meant for testing |
| Cloudflare Workers AI | $0.30 / $0.10 / $1.00 | Not on Workers Free | Zero data retention, function calling, runs inside Workers |
| Self-host (NVFP4) | GPU time instead of tokens | Weights are free under Apache 2.0 | Needs 500GB+ VRAM (2x B300 or 4x H200) |
The headline input and output prices are identical everywhere I checked, so the real difference is the cached-input rate. For an agent run with 2M input tokens, 80% of them prompt-cache hits, and 100K output tokens, my calculation gives $0.268 on Together or Modal, $0.316 on Nebius and $0.380 on Cloudflare. With no cache hits, the same job costs $0.70 on all four. If your workload resends long system prompts or repository context, the cache rate matters more than anything else on the page.
When self-hosting actually pays off
To put the hardware requirement in money terms, I used Modal's public per-second GPU prices as a reference point: $0.001972/s for a B300, $0.001736/s for a B200 and $0.001261/s for an H200. That makes the smallest official NVFP4 setup, 2x B300, about $14.20 per hour, or roughly $10,223 for a month running around the clock. 4x H200 comes to about $18.16 per hour, and the vLLM recipe's 4x B200 setup to about $25.00.
At $1.00 per million output tokens, $14.20 buys 14.2 million output tokens from an API. So a 2x B300 server only beats the API on output cost if it produces more than about 3,944 output tokens per second, every second it is running, before counting storage, idle time and your own engineering hours. The vLLM recipe reports around 1,100 output tokens per second per GPU on GB200 with DSpark at 100 tokens per second per user, which shows high throughput is possible with many concurrent users, but that is a different GPU and a batch-heavy setting. For a single developer or a small team with bursty traffic, the API wins easily. Self-hosting makes sense when you already own the GPUs, need data to stay on your own hardware, want to serve your own fine-tuned weights, or have steady traffic high enough to keep a node busy.
Editor's note
Good fit: teams building coding agents or document and chart workflows who want an Apache 2.0 model they can later fine-tune or move in-house; anyone already running vLLM or SGLang on B200/B300/H200 nodes.
Not a fit: local hobby setups on consumer GPUs or a single workstation. The official minimum is 500GB of VRAM, and I found no Inkling 2 Small low-bit GGUF yet. Also not ideal as a polished chatbot: Thinking Machines itself says the model is less optimized for chat, is not the most succinct communicator, and does not support audio.
Gotchas I ran into while checking: Cloudflare's model page says 18B active parameters while Thinking Machines, Hugging Face, Together and Modal all say 17B; I went with the vendor's 17B. Hugging Face's sidebar shows "383B params" for the NVFP4 repository and "612B" for MXFP8, which most likely reflects how packed low-bit weights are counted in the file metadata, not a different model. The announcement's API partner links for Baseten and NVIDIA did not resolve to model pages when I tried them.
Common errors and fixes: SGLang fails at load with MegaMoE enabled: your DeepGEMM build lacks Inkling 2 Small support, so switch to routed experts. Long prompts rejected by vLLM: the recipe's 4-GPU command sets --max-model-len 262144; the recipe only verified the full 1M window with TP8 across two GB200 trays. Expert LoRA adapters fail on the shipped checkpoint: the vLLM recipe says expert adapters need BF16 expert weights, so use attention-only adapters. AMD launch refused: only MI350X/MI355X are validated.
FAQ
Is Inkling 2 Small really open source?
The weights are published on Hugging Face under the Apache 2.0 license, which allows commercial use and fine-tuning. Training data is not released.
Can I run Inkling 2 Small on one GPU or a Mac?
Not with the official checkpoints. The smallest configuration in the model card is 2x NVIDIA B300 for NVFP4. A community or Unsloth low-bit quantization could change that later, but none was available from Unsloth when I checked.
Which API is cheapest?
All four providers I verified charge $0.30 per million input tokens and $1.00 per million output tokens. Together and Modal have the lowest cached-input rate at $0.03, so they come out cheapest for cache-heavy agent workloads.
Related on lightmind24: vLLM Errors Fixed: Out of Memory, CUDA and Chat Templates · Modal vs Runpod vs Lambda: What an H100 Hour Really Costs · Together AI Pricing: Credits, Rates and Limits
Start on an API, measure your real token volume for a few weeks, and only then decide whether a GPU node is worth it.
Screenshots: see the official announcement update, coding benchmark table, Hugging Face hardware requirements and Cloudflare model info, plus my own self-host vs API cost table, at the end of this post. Images: thinkingmachines.ai, huggingface.co, developers.cloudflare.com, and lightmind24 own calculation table.
Last verified: 2026-11-13
Sources: Thinking Machines: Introducing Inkling 2 Small · Hugging Face: Inkling-2-Small model card · Inkling-2-Small-MXFP8 · vLLM recipe · SGLang cookbook · Unsloth Inkling docs · Together AI model page · Modal model page · Modal pricing · Nebius Token Factory endpoint · Cloudflare Workers AI model page · Tinker models and pricing
lightmind24 is an independent review site, not affiliated with Thinking Machines, Together AI, Modal, Nebius or Cloudflare.
댓글
댓글 쓰기