기본 콘텐츠로 건너뛰기

Inkling 2 Small: Run It Locally or via API, Which Route Fits

In short: Thinking Machines released the weights of Inkling 2 Small on Hugging Face on November 12, 2026 under Apache 2.0. It is a 594B-parameter Mixture-of-Experts model with 17B active parameters, so "running it locally" means a multi-GPU server with at least 500GB of combined VRAM, not a laptop. For most people the hosted APIs at $0.30 per million input tokens and $1.00 per million output tokens are the practical route; self-hosting only pays off at very high, steady volume or when the data cannot leave your own hardware.

What Inkling 2 Small is, in numbers

Inkling 2 Small is the first model of Thinking Machines' second Inkling generation. The facts that decide which route fits, all checked on the official announcement and Hugging Face model card this morning:

  • Size: 594B total parameters, 17B active per token. Each token is routed to 8 of 384 experts across 48 decoder layers.
  • Inputs and outputs: text and image in, text out. There is no audio input, unlike the first-generation Inkling models.
  • Context: up to 1M tokens (1,048,576) according to the model card and the vLLM recipe.
  • Reasoning effort: variable. Thinking Machines ran its own evaluations at effort 0.99 ("xhigh").
  • License: Apache 2.0, shown on the Hugging Face model page.
  • Checkpoints: the main thinkingmachines/Inkling-2-Small repository in NVFP4 (the precision the model was trained in), and thinkingmachines/Inkling-2-Small-MXFP8. The model card recommends NVFP4 for all use cases and calls the 1–2 point gain of MXFP8 on evals not meaningful in practice.

The announcement was first published on November 5 with Tinker-only access; the November 12 update added the open weights plus integrations with SGLang, vLLM, TokenSpeed and Unsloth, and a list of API partners.

Route 1: Download the weights and self-host

Check the hardware first

The model card is blunt about it. The NVFP4 checkpoint needs a GPU cluster with at least 500GB of aggregated VRAM, for example 2x NVIDIA B300 or 4x NVIDIA H200. The MXFP8 checkpoint needs at least 800GB, for example 4x B300 or 8x H200. I queried the Hugging Face file API to see what you actually download: the NVFP4 repository holds 393.5 GB of safetensors in 25 shards, and the MXFP8 repository holds 622.8 GB in 49 shards. Budget disk space and download time accordingly, and keep both on fast local storage rather than a network mount.

The SGLang cookbook adds useful detail: at tensor parallel 4 the NVFP4 weights take about 94 GB per GPU and the MXFP8 weights about 151 GB per GPU, which is why 192GB cards such as B200 run the MXFP8 version across a full 8-GPU node.

Option A: vLLM

The official vLLM recipe (updated November 12) says to use the dedicated vllm/vllm-openai:inkling-2-small Docker image until support lands in a regular vLLM release. Its default configuration is NVFP4 on 4x B200 with tensor parallel 4:

docker run --gpus all --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:inkling-2-small thinkingmachines/Inkling-2-Small \
  --tensor-parallel-size 4 \
  --max-model-len 262144 \
  --max-num-seqs 256 \
  --reasoning-parser inkling \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'

Notes from the recipe that matter in practice: one GB200 tray at TP4 was verified for contexts up to 262,144 tokens, and you need two trays (TP8) for the full 1M window. The checkpoint ships with a bundled DSpark speculative-decoding draft. Tool calling uses the inkling tool-call parser, which the recipe says is unit-tested but not yet benchmarked end to end. Once the server is up, it speaks the OpenAI-compatible chat API on port 8000.

Option B: SGLang

The SGLang cookbook has verified launch commands for GB300, GB200, B300, B200, H200, AMD MI350X/MI355X, Vera Rubin and TPU7x. The Blackwell recipes run TP4/EP4; H200 and AMD run TP8/EP8. A trimmed version of the GB300 NVFP4 command:

sglang serve \
  --model-path thinkingmachines/Inkling-2-Small \
  --tp 4 --ep 4 \
  --moe-a2a-backend megamoe \
  --attention-backend fa4 \
  --enable-multimodal \
  --reasoning-parser inkling \
  --tool-call-parser inkling2 \
  --host 0.0.0.0 --port 30000

Two gotchas from the cookbook: keep EP equal to TP, because the NVFP4 experts cannot be tensor-sharded, and the MegaMoE backend needs a DeepGEMM build with Inkling 2 Small support, otherwise the server fails at load. If you do not have that build, pick the routed-experts backend instead. On AMD, only gfx950 GPUs (MI350X/MI355X) are validated; other ROCm GPUs refuse at launch.

Option C: Unsloth and smaller quantizations

The announcement and model card both list Unsloth as an integration, and Unsloth has published low-bit GGUF versions of the first-generation Inkling models before. But when I checked this run, Unsloth's Inkling docs page still covered only Inkling and Inkling-Small, and I could not find a public Inkling 2 Small GGUF repository under Unsloth's Hugging Face organization. If you were hoping for a Mac Studio-sized 1- or 2-bit build, treat that as "not yet" and check back, rather than assuming the first-generation memory table applies.

Route 2: Use a hosted API

The announcement lists seven API partners: NVIDIA, Fireworks, Together, Baseten, Nebius, Modal and Cloudflare. I could confirm a live model page with prices for four of them, plus Thinking Machines' own Tinker. The Baseten and NVIDIA model URLs linked from the announcement returned "not found" pages when I checked, and I could not find Inkling 2 Small on Fireworks' public model or pricing pages, so they are left out of the table below rather than guessed.

  • Together AI: model thinkingmachines/Inkling-2-Small, serverless and dedicated, OpenAI-style chat completions endpoint.
  • Modal: a shared, OpenAI-compatible endpoint billed per token; Modal's page shows the one-line modal endpoint create --shared --model thinkingmachines/inkling-2-small setup.
  • Nebius Token Factory: a public shared endpoint serving an FP4 quantization, with tool calling, reasoning and Responses API support. Nebius itself says the shared endpoint is meant for tests, not production, and that its region may change.
  • Cloudflare Workers AI: model ID @cf/thinkingmachines/inkling-2-small, with function calling, vision, reasoning levels from none to max, and zero data retention. It is not available on the Workers Free plan; you need Workers Paid or prepaid AI Gateway credits.
  • Tinker: serverless inference is in beta, and Thinking Machines runs a waitlist for production use. Tinker is also where you fine-tune the model ($1.50 per million training tokens at 64K context, $3.00 at 256K).

Comparison: which hosted route costs what

Route Input / cached / output (per 1M tokens) Free tier What stands out
Together AI $0.30 / $0.03 / $1.00 None listed for this model Serverless and dedicated; 1M context
Modal shared endpoint $0.30 / $0.03 / $1.00 None listed for this model Pay per token, OpenAI-compatible; option to deploy your own GPUs on the same platform
Nebius Token Factory $0.30 / $0.06 / $1.00 None listed for this model FP4 quantization, ~130 tok/s shown; shared endpoint meant for testing
Cloudflare Workers AI $0.30 / $0.10 / $1.00 Not on Workers Free Zero data retention, function calling, runs inside Workers
Self-host (NVFP4) GPU time instead of tokens Weights are free under Apache 2.0 Needs 500GB+ VRAM (2x B300 or 4x H200)

The headline input and output prices are identical everywhere I checked, so the real difference is the cached-input rate. For an agent run with 2M input tokens, 80% of them prompt-cache hits, and 100K output tokens, my calculation gives $0.268 on Together or Modal, $0.316 on Nebius and $0.380 on Cloudflare. With no cache hits, the same job costs $0.70 on all four. If your workload resends long system prompts or repository context, the cache rate matters more than anything else on the page.

When self-hosting actually pays off

To put the hardware requirement in money terms, I used Modal's public per-second GPU prices as a reference point: $0.001972/s for a B300, $0.001736/s for a B200 and $0.001261/s for an H200. That makes the smallest official NVFP4 setup, 2x B300, about $14.20 per hour, or roughly $10,223 for a month running around the clock. 4x H200 comes to about $18.16 per hour, and the vLLM recipe's 4x B200 setup to about $25.00.

At $1.00 per million output tokens, $14.20 buys 14.2 million output tokens from an API. So a 2x B300 server only beats the API on output cost if it produces more than about 3,944 output tokens per second, every second it is running, before counting storage, idle time and your own engineering hours. The vLLM recipe reports around 1,100 output tokens per second per GPU on GB200 with DSpark at 100 tokens per second per user, which shows high throughput is possible with many concurrent users, but that is a different GPU and a batch-heavy setting. For a single developer or a small team with bursty traffic, the API wins easily. Self-hosting makes sense when you already own the GPUs, need data to stay on your own hardware, want to serve your own fine-tuned weights, or have steady traffic high enough to keep a node busy.

Editor's note

Good fit: teams building coding agents or document and chart workflows who want an Apache 2.0 model they can later fine-tune or move in-house; anyone already running vLLM or SGLang on B200/B300/H200 nodes.

Not a fit: local hobby setups on consumer GPUs or a single workstation. The official minimum is 500GB of VRAM, and I found no Inkling 2 Small low-bit GGUF yet. Also not ideal as a polished chatbot: Thinking Machines itself says the model is less optimized for chat, is not the most succinct communicator, and does not support audio.

Gotchas I ran into while checking: Cloudflare's model page says 18B active parameters while Thinking Machines, Hugging Face, Together and Modal all say 17B; I went with the vendor's 17B. Hugging Face's sidebar shows "383B params" for the NVFP4 repository and "612B" for MXFP8, which most likely reflects how packed low-bit weights are counted in the file metadata, not a different model. The announcement's API partner links for Baseten and NVIDIA did not resolve to model pages when I tried them.

Common errors and fixes: SGLang fails at load with MegaMoE enabled: your DeepGEMM build lacks Inkling 2 Small support, so switch to routed experts. Long prompts rejected by vLLM: the recipe's 4-GPU command sets --max-model-len 262144; the recipe only verified the full 1M window with TP8 across two GB200 trays. Expert LoRA adapters fail on the shipped checkpoint: the vLLM recipe says expert adapters need BF16 expert weights, so use attention-only adapters. AMD launch refused: only MI350X/MI355X are validated.

FAQ

Is Inkling 2 Small really open source?

The weights are published on Hugging Face under the Apache 2.0 license, which allows commercial use and fine-tuning. Training data is not released.

Can I run Inkling 2 Small on one GPU or a Mac?

Not with the official checkpoints. The smallest configuration in the model card is 2x NVIDIA B300 for NVFP4. A community or Unsloth low-bit quantization could change that later, but none was available from Unsloth when I checked.

Which API is cheapest?

All four providers I verified charge $0.30 per million input tokens and $1.00 per million output tokens. Together and Modal have the lowest cached-input rate at $0.03, so they come out cheapest for cache-heavy agent workloads.

Related on lightmind24: vLLM Errors Fixed: Out of Memory, CUDA and Chat Templates · Modal vs Runpod vs Lambda: What an H100 Hour Really Costs · Together AI Pricing: Credits, Rates and Limits

Start on an API, measure your real token volume for a few weeks, and only then decide whether a GPU node is worth it.

Screenshots: see the official announcement update, coding benchmark table, Hugging Face hardware requirements and Cloudflare model info, plus my own self-host vs API cost table, at the end of this post. Images: thinkingmachines.ai, huggingface.co, developers.cloudflare.com, and lightmind24 own calculation table.

Last verified: 2026-11-13

Sources: Thinking Machines: Introducing Inkling 2 Small · Hugging Face: Inkling-2-Small model card · Inkling-2-Small-MXFP8 · vLLM recipe · SGLang cookbook · Unsloth Inkling docs · Together AI model page · Modal model page · Modal pricing · Nebius Token Factory endpoint · Cloudflare Workers AI model page · Tinker models and pricing

lightmind24 is an independent review site, not affiliated with Thinking Machines, Together AI, Modal, Nebius or Cloudflare.

Uploaded Image Uploaded Image Uploaded Image Uploaded Image

댓글

이 블로그의 인기 게시물

브루(Vrew) 요금제 정리: 무료 200크레딧으로 어디까지 쓸 수 있나

요약: 영상 자막 편집기 브루(Vrew)는 2026년 4월 22일부터 기능별 사용량 대신 '통합 크레딧' 하나로 요금을 매겨요. 무료 플랜은 매달 200크레딧이 들어오는데, 음성 분석(자동 자막)만 쓰면 최대 200분 분량이라 짧은 영상 몇 편은 충분하지만, 영상 초반의 Vrew 마크를 지우려면 유료 플랜이 필요해요. 이 글은 브루 공식 가격 페이지와 고객센터 공지를 2026년 10월 23일에 다시 확인해, 크레딧 계산법과 플랜 고르는 기준을 정리했어요. 목차 브루 요금 구조: 통합 크레딧이란 1크레딧으로 할 수 있는 일 무료 플랜으로 되는 것과 안 되는 것 유료 플랜 4종 비교표 월 구독·연 구독·기간 이용권, 무엇이 이득일까 상황별 추천: 학생·직장인·유튜버 크레딧이 모자랄 때: 사용 기록 지우개 업그레이드·환불 전에 알아둘 규칙 운영자 메모, 자주 묻는 질문 1. 브루 요금 구조: 통합 크레딧이란 예전 브루는 플랜마다 음성 분석 몇 분, AI 목소리 몇 자, 번역 몇 자처럼 기능별 한도를 따로 줬어요. 그런데 보이저엑스(브루 운영사)가 실제 이용 데이터를 보니 한두 가지 기능만 집중해서 쓰는 사람이 훨씬 많았다고 해요. 그래서 2026년 4월 22일 업데이트부터는 플랜마다 정해진 크레딧을 받고, 어떤 기능에 쓸지는 내가 정하는 방식으로 바뀌었어요. 플랜별 매달 지급량은 Free 200 · Light 2,000 · Standard 10,000 · Business 12,000크레딧 이에요. 공식 공지에 따르면 무료 사용자는 매달 1일에, 유료 사용자는 매월 결제일에 충전되고, 남은 크레딧은 다음 달로 넘어가지 않아요. 바뀐 뒤 무엇이 좋아졌나 공지에 나온 예시가 이해하기 쉬워요. Standard 플랜에서 AI 목소리만 쓰면 예전에는 최대 50만 자였지만, 지금은 1만 크레딧을 모두 AI 목소리에 몰아 쓸 수 있어 최대 100만 자가 돼요. 반대로 여러 기능을 골고루 쓰는 사람은 크레딧이 기능끼리 ...

에이닷 전화 vs 익시오: 통화 녹음·AI 요약, 내 통신사·폰에 맞는 앱 고르기

요약: 통화 녹음과 AI 요약을 쓰고 싶다면 먼저 내 통신사와 휴대폰 종류부터 확인해야 해요. LG유플러스 고객은 무료 앱 익시오, SKT 고객은 에이닷 전화가 가장 기능이 많고, 다른 통신사의 아이폰 사용자는 iOS에 들어 있는 기본 통화 녹음·전사 기능이 현실적인 선택이에요. 이 글은 2026-10-30에 SKT·LG유플러스·Apple 공식 페이지와 도움말을 직접 확인해 세 가지를 비교했어요. 왜 '앱 기능'보다 '내 조건'을 먼저 봐야 할까요 통화 요약 앱 비교 글을 보면 기능 목록부터 나열하는 경우가 많아요. 그런데 실제로 써 보려는 사람에게 가장 먼저 걸리는 건 기능이 아니라 이용 조건 이에요. 익시오는 LG유플러스 고객 전용이고, 에이닷 전화는 누구나 설치할 수 있지만 통역콜·baro 같은 기능은 SKT 고객에게만 열려 있어요. 아이폰은 운영체제 구조상 앱이 통화를 마음대로 녹음하기 어려워서, 통신사 앱이 안 되면 iOS 기본 기능에 기대야 해요. 그래서 이 글은 "어떤 앱이 제일 좋은가"보다 "내 통신사·내 폰에서 무엇을 쓸 수 있나" 순서로 정리했어요. 기능 차이는 그다음이에요. 한눈에 비교: 에이닷 전화 vs 익시오 vs 아이폰 기본 통화 녹음 구분 에이닷 전화 (SKT) 익시오 ixi-O (LG유플러스) 아이폰 기본 통화 녹음 (Apple) 누가 쓸 수 있나 통신사 구분 없이 앱 설치 가능. 단, 통역콜·baro·그룹통화 발신 등은 SKT 고객 전용 U+ 모바일 고객. 타 통신사, 만 14세 미만, 선불요금제 고객은 제한 (공식 사이트 FAQ상 알뜰폰 중 KB리브엠은 예외) 지원 국가·언어의 iPhone 사용자. 1명과 통화할 때만 녹음 가능 지원 OS·기기 Android 10 이상, iOS 16 이상 iOS 17 이상 iPhone 12 이상 / Android...

타입캐스트 vs 클로바더빙: 수익 나는 채널에 쓸 때 무료·유료 조건 비교

요약 타입캐스트와 클로바더빙은 둘 다 글만 입력하면 한국어 AI 목소리를 만들어 주는 서비스지만, 무료로 쓸 수 있는 범위가 꽤 다릅니다. 클로바더빙 무료는 매달 15,000자까지 쓸 수 있는 대신 수익이 생기는 채널에는 쓸 수 없고, 타입캐스트 무료는 처음에 주는 약 5분 분량의 다운로드로 끝나는 체험판에 가깝습니다. 광고 수익이 붙은 유튜브나 쇼핑몰 영상에 넣을 목소리라면 처음부터 유료 플랜 조건을 보고 고르는 편이 안전합니다. 왜 이 두 서비스를 같이 비교하나 쇼츠 내레이션, 수업 자료 더빙, 상품 소개 영상처럼 "내 목소리 대신 AI 목소리"가 필요할 때 한국 사용자들이 가장 많이 찾는 곳이 네오사피엔스의 타입캐스트 와 네이버의 클로바더빙 입니다. 기능 소개만 보면 비슷해 보이지만, 실제로 문제가 되는 건 음질보다 "이 목소리를 내 채널에 써도 되나" 입니다. 그래서 이 글은 기능 나열보다 무료 사용 범위, 출처 표기 의무, 유료 플랜 가격과 다운로드 한도를 중심으로 정리했습니다. 모든 수치는 2026년 10월 25일에 두 서비스의 공식 요금 페이지와 고객센터 도움말에서 직접 다시 확인한 내용입니다. 한눈에 보는 비교표 항목 타입캐스트 무료 타입캐스트 베이직 클로바더빙 Free 클로바더빙 Standard 월 요금 0원 9,900원 (연간 결제 시 10% 할인) 0원 19,900원 (네이버 클라우드 플랫폼에서 결제) 무료·포함 사용량 음성 생성·재생 무제한, 다운로드는 최초 3,000크레딧(약 5분) 1회 지급 월 30,000크레딧(약 60분) 월 15,000자, 다운로드 월 20회, 프로젝트 누적 5개 (매월 1일 갱신) 월 30,000자, 다운로드 월 40회, 프로젝트 누적 20개 쓸 수 있...