LightOn AI has released ColGrep-Agent-2B, an open 2-billion-parameter code-search agent that runs on your machine and sits between a frontier coding agent and the repository. Instead of making Claude (or another paid model) spend several turns grepping, opening files and chasing references, the small local agent does that exploration and hands back file paths, line ranges and the code itself. The company published the write-up on October 12 on the Hugging Face blog.
In LightOn’s own tests with Claude Code and Sonnet 5.5, delegating search to ColGrep-Agent cut API tokens by 34% and cache-aware API cost by 40% compared with plain grep, with no statistically significant change in answer accuracy on their code-search questions.
What was released
There are three related pieces, and the blog is careful to keep them separate:
- ColGrep is the semantic code search engine (built on NextPlaid and a 17M-parameter multi-vector retriever called LateOn-Code-edge).
- ColGrep-Agent-2B is the 2B model that decides what to search, what to open and when to stop.
colgrep --agentis the command that runs the model with ColGrep and a read-only terminal.
Weights ship as GGUF (the default install pulls colgrep-agent-2B-Q8_0.gguf, about 2.5 GB). There is no Python environment to set up and no API key. Inference uses the GPU when available—Metal on Apple Silicon, Vulkan on NVIDIA, AMD and Intel—or falls back to the CPU. On an M3 Pro, LightOn reports roughly 20 seconds per question with Metal and 3–4 GB of peak memory.
The agent has three tools only: colgrep for semantic search (with optional regex and path filters), a read-only terminal for inspecting files, and finish to return locations as path/to/file:START-END. It is capped at 10 turns, and the harness forces a finish call on the last turn. Installers cover Homebrew, a shell script, Windows PowerShell and Cargo, plus one-liners to hook Claude Code, Codex and OpenCode.
The key numbers LightOn reports
The headline experiment used Claude Code with Sonnet 5.5 on 52 code-search questions across 13 public repositories created in 2026 (Rust, Go, TypeScript and Python, from about 17k to 1.8M lines). Each question ran three times in each of four setups—624 sessions total:
- Grep: built-in grep, find and file reads only
- ColGrep: Claude can call semantic search directly
- ColGrep-Agent: Claude delegates searches to
colgrep --agent - Balanced: Claude can use plain ColGrep or the agent
Against Grep’s baseline of about 142k tokens and 7.3¢ cache-aware cost per question, ColGrep-Agent used about 93k tokens (34.3% fewer; 95% CI 28–40%) and 4.4¢. It used fewer tokens on 88% of questions. Correct answers were 80.8% for ColGrep-Agent versus 83.3% for Grep; LightOn says the −2.6 percentage-point gap is not statistically significant (95% CI −10.9 to +5.1). The Balanced setup matched Grep’s observed 83.3% accuracy while saving 21% of tokens.
The trade-off is time. End-to-end question time rose from 13.0 s with Grep to 21.6 s with ColGrep-Agent on their GPU server. Turns fell from 5.5 to 3.9 on average. Savings grew with repository size: about 19% below 50k lines, 29% from 50k to 150k, and 40% from 150k to 600k.
On separate localization benchmarks (SWE-bench Lite and Multi-SWE-bench Flash, adapted to “find the files and lines,” not generate a patch), ColGrep-Agent-2B scored 76.18 file F1 / 19.92 line F1 on SWE-bench Lite and 62.57 / 15.93 on Multi-SWE-bench Flash—competitive with much larger open models in the same harness, and the highest line F1 among the models LightOn listed on SWE-bench Lite. On the Semble code-search benchmark it returned 1.7 files on average with Precision@10 of 0.744 and F1 of 0.776.
Why it matters
Frontier coding agents are expensive partly because exploration is chatty. Every extra grep turn resends a growing context—about 23k tokens per turn in LightOn’s runs. Pushing that first pass of “where is this?” onto a local specialist is a concrete way to cut billed tokens without throwing away the big model for the hard reasoning.
It also fits a broader pattern: small, task-trained models for the repetitive middle of a workflow, frontier models for the judgment call. LightOn started from MiniCPM5-2B, did a short SFT cold start from Qwen3.5-27B trajectories, then reinforcement learning with a GRPO variant. The released colgrep --agent command uses the same prompt, tools and turn budget as training.
Caveats are worth keeping in view. The Claude Code results are for code-search questions, not full end-to-end coding tasks. Accuracy confidence intervals are wide. Latency goes up. And LightOn’s own comment thread notes that when Claude only sees what the agent returns, a missed file can stay invisible downstream—so the Balanced hook mode may be the safer default for many teams.
What you can do
- Install ColGrep and the agent (
brew install lightonai/tap/colgrep && colgrep --install-agent, or the shell/Windows/Cargo options in the post), then runcolgrep initin a repo. - Try
colgrep --agent "where is X configured"on a question you would normally dump into a frontier agent. - If you use Claude Code, Codex or OpenCode, install the integration and compare
hook-mode agent(max savings) withhybrid(Balanced) on a few real repo questions, watching both token use and whether locations are complete. - Treat the 34% and 40% figures as LightOn’s measured result on their 52-question set—not a guarantee for your codebase—and judge it on your own repos and latency budget.
댓글
댓글 쓰기