기본 콘텐츠로 건너뛰기

Agent Initiative Benchmark: GPT-6 vs Claude on Staying Quiet

Summary: Deepnote's Agent Initiative Benchmark tests something most model leaderboards skip: whether an AI teammate in Slack can tell when to step in and when to say nothing. Across 12 models and 50 workplace situations, GPT-6 Astra (8.97) and GPT-6.1 Sol (8.65) lead, Claude Fable 5.2 is third at 8.17, and Sol does it for about one sixth of Astra's cost per task. The bigger lesson for anyone building a proactive agent: almost every model is good at helping and noticeably worse at staying quiet.

What the benchmark measures

Most agent benchmarks hand the model a task and grade the answer. This one hands the model a company and waits. The agent sits inside the Slack workspace of PageMantle, a synthetic software company with 71 employees and 100 enterprise customers, and receives one standing instruction: assess what is going on and help if it is useful. Nobody tags it.

From there it has three jobs. It has to dig up evidence in systems the conversation never mentions, decide whether a message is needed at all, and, if so, pick the right audience: a channel post, a thread reply or a direct message. In 10 of the 50 situations the correct move is silence, because the issue has already been handled and the records say so.

Deepnote's example makes the idea concrete. Sales says in one channel that a launch is on Friday; engineering says in another that the release slipped a week. A useful agent spots the mismatch without being asked, finds the release update and brings it into the sales thread.

The setup in numbers

  • 11 simulated services: Slack, Linear, Notion, HubSpot, Stripe, Google Calendar, PostHog, Google Analytics, Google Search Console, BigQuery and Snowflake, each behind an API shaped like the real one.
  • About 10 GB of data covering six years of company history, with one live month of Slack activity (roughly 9k messages and 2.5k threads).
  • 50 tasks, 8 task types: Find, Explain, Analyze, Verify, Decide, Diagnose, Coordinate and Guard.
  • Same conditions for every model: identical instructions and tools, high reasoning effort, and a budget of 180 seconds of model response time (tool execution doesn't count). A run that hits the limit without sending a needed message scores as a miss.
  • Grading: two LLM judges, blind to which model they are scoring, rate each run 0 to 10 against a reference solution on four things: the decision to intervene, content and evidence, clarity and tone, and audience and location.

The headline results

Here is how the field lines up on the dimensions that matter when you pay for an always-on agent. All figures come from Deepnote's results page; the "month of Slack" column is Deepnote's own estimate of one investigation per thread across about 2,500 threads.

Model Score (0-10) Helped when needed Stayed silent when right Cost per task Avg. response time Month of Slack
GPT-6 Astra 8.97 40/40 8/10 $0.693 54 s $1,732
GPT-6.1 Sol 8.65 39/40 8/10 $0.112 54 s $281
Claude Fable 5.2 8.17 40/40 6/10 $1.87 1 min 0 s $4,684
Claude Opus 5.5 7.84 40/40 8/10 $1.28 29 s $3,196
GPT-6 Luna 6.38 38/40 4/10 $0.0070 50 s $18

Three things stand out once you read past the ranking.

1. Astra and Sol are statistically tied

Sol's 8.65 sits inside Astra's 95% interval (8.30 to 9.52), so Deepnote itself says this task set does not separate them. If you are choosing between the two, the deciding factor is price: $0.112 versus $0.693 per task.

2. The priciest models are not the best ones

The three most expensive models per task (Fable 5.1 at $3.02, Fable 5.2 at $1.87 and Opus 5.5 at $1.28) finish third, fourth and fifth. GPT-6 Luna and Claude Fable 5.1 are about 430 times apart in price and only 1.1 points apart in score.

3. More tool calls did not mean better answers

Gemini 4 Argon averaged 34.5 tool calls per task and Grok 4.7 averaged 35.5, the most in the field, yet they scored lowest (5.60 and 4.51). Both kept investigating until the clock ran out: 27 and 18 of their 50 runs hit the time limit before any message went out, and they scored zero on 17 and 20 tasks. Meanwhile Opus 5.5 answered in about 29 seconds with 10 tool calls.

Why "knowing when not to act" is the real test

When help was needed, models sent a message about 9 times out of 10, and nearly every miss was a timeout rather than a deliberate choice. When silence was right, they held back only about 6 times out of 10. Even the best (Astra, Sol and Opus 5.5) still spoke up in 2 of the 10 settled situations. GPT-6 Luna, DeepSeek V4.1 Flash and Mistral Large 4 spoke in 6 of 10.

The uncomfortable detail is that those unnecessary messages were accurate and well placed. The matter was simply already closed. So a filter that checks message quality won't catch them; you need a separate check for "is this already handled?". Deepnote also found that speaking out of turn cost about 1.8 times as much as staying quiet for the same model, and took about 1.8 times as long, so false alarms hurt both your team's attention and your bill.

Gemini 4 Argon is a useful warning about reading one number in isolation. It stayed silent correctly in 10 of 10 cases, the best in the field, but it sent a message in only 23 of the 40 situations that needed one. Perfect restraint that comes from running out of time is not judgment.

Strengths by task type

Rankings shift by task type, and each type has only 5 to 8 tasks, so treat these as tendencies to test on your own work rather than fixed rules:

  • Analyze is the hard one. Pulling numbers from PostHog, BigQuery or Snowflake and checking them against a claim in Slack is the weakest type for 8 of 12 models (field average 5.17). Astra (8.44) and Sol (8.38) are the clear exceptions.
  • Verify is the easy one. Checking a claim against its sources averages 8.00 across all models.
  • Astra leads five of eight types, including Coordinate (9.33) and Guard (9.83).
  • Sol leads Explain (9.08) and Find (8.60) but is about 3 points behind Astra on Coordinate.
  • Fable 5.2 has the top Decide score (9.58), with Sol (9.50) and Astra (9.42) close behind.

How to choose a model for your own proactive agent

Putting the numbers together, this is how we would approach the choice. It is our reading of Deepnote's data, not a recommendation from Deepnote, and the benchmark tested models, not routers.

  1. Start from your task mix. List what people would actually ask the agent for. A support team that mostly needs "who owns this?" (Find) has different needs from a finance team reconciling numbers (Analyze).
  2. Use a cheap capable model as the default watcher. Sol gives you near-top scores at $281 for a month of Slack in Deepnote's estimate. Luna and DeepSeek V4.1 Flash are far cheaper but held back in only 4 of 10 silence cases, so use them for triage, not the final decision to post.
  3. Escalate the hard cases. Numbers to reconcile, a decision to weigh or a thread that may already be closed are where Astra's extra cost pays off.
  4. Pick speed for live threads, depth for background work. Someone waiting in a thread benefits from Opus 5.5's roughly 29-second replies; a nightly digest can afford Sol's 54 seconds and deeper digging.
  5. Cap the search. Give your harness a tool-call or time budget and a point where it must commit to a message or to silence.
  6. Track false alarms as their own metric, separately from answer quality.

Run it against your own model

The code is on GitHub under Apache 2.0 and the PageMantle dataset is on Hugging Face under CC BY 4.0. According to the repository's quickstart, you need Python 3.13 on Linux or macOS, plus a C++ compiler for the hnswlib dependency. The flow is:

  1. Clone the repository, create a Python 3.13 virtual environment and install from requirements.lock, then install the package in editable mode.
  2. Run benchmark init. It creates benchmark.json, where you set one agent model and two different judge models. Each profile names an API format (OpenAI Responses, OpenAI-compatible Chat Completions, Anthropic Messages, or a custom Python adapter), an endpoint and a model ID. The model must support function tools.
  3. Run benchmark data download, then benchmark check, which validates settings without calling any model.
  4. Run benchmark run --task task-01 for one example, or --task all for all eight bundled examples (one per task type), and benchmark results to see the scores.

The docs also note that changing the model, API protocol or generation settings counts as a different experiment, so keep everything fixed when you compare.

Editor's note

  • Who it's for: teams building or buying a Slack-style agent that watches conversations and decides on its own when to chime in.
  • Not for: picking a coding model or a chat assistant you prompt directly. This benchmark doesn't measure either.
  • Gotcha 1, one effort level: every model ran at high reasoning effort. Deepnote says it has not run lower or higher effort sweeps, and both score and cost would move.
  • Gotcha 2, costs are historical: the results page says recorded costs are observations, not live price quotes. Recompute with current prices before you budget.
  • Gotcha 3, small sample per type: 50 tasks overall, 5 to 8 per type. The overall ranking is more reliable than any single type.
  • Gotcha 4, Slack only: agents reply only through Slack in this version; email, documents and actions beyond messaging are out of scope.
  • Common setup error: the install fails building hnswlib if Python 3.13 development headers or a C++ compiler are missing. The quickstart points to xcode-select --install on macOS and build-essential plus Python 3.13 headers on Ubuntu.
  • Common config error: using the same model ID for both judges. The docs say the two judge model IDs must differ.

FAQ

Which model won the Agent Initiative Benchmark?

GPT-6 Astra had the highest mean score at 8.97, but GPT-6.1 Sol (8.65) falls inside Astra's confidence interval, so Deepnote does not treat them as separated. Claude Fable 5.2 is third at 8.17.

What does "staying silent" mean in this benchmark?

In 10 of the 50 situations the problem had already been handled and the records showed it. The correct action was to send nothing. Models were judged on whether they held back, not just on whether their message was good.

Can I test my own agent or a local model?

Yes. The harness accepts OpenAI Responses, OpenAI-compatible Chat Completions and Anthropic Messages endpoints, or a custom Python adapter, and the docs describe a setting for a local server without authentication. The model must support function tools.

Related on lightmind24: When a Small Decider Beats a Frontier LLM: Strands Decider · Mem0 vs Zep vs Letta: Choosing an AI Agent Memory Layer · AI by Zapier Playbook: Sort Form Leads and Alert Slack

If you are building an agent that lives in your team's chat, test the moments it should stay out of as carefully as the ones where it should help.

Screenshots of Deepnote's results page and our own model-fit table are at the end of this post. Screenshots: deepnote.com/research/benchmark, captured 2026-11-14 (vendor icons removed); model-fit table by lightmind24.

Last verified: 2026-11-14
Sources: Deepnote Research: Agent Initiative Benchmark · GitHub: deepnote/agent-initiative-benchmark · Benchmark quickstart · Benchmark model settings · PageMantle dataset on Hugging Face

lightmind24 is an independent review site, not affiliated with Deepnote, OpenAI, Anthropic or any other lab named here.

Uploaded Image Uploaded Image Uploaded Image Uploaded Image Uploaded Image

댓글

이 블로그의 인기 게시물

브루(Vrew) 요금제 정리: 무료 200크레딧으로 어디까지 쓸 수 있나

요약: 영상 자막 편집기 브루(Vrew)는 2026년 4월 22일부터 기능별 사용량 대신 '통합 크레딧' 하나로 요금을 매겨요. 무료 플랜은 매달 200크레딧이 들어오는데, 음성 분석(자동 자막)만 쓰면 최대 200분 분량이라 짧은 영상 몇 편은 충분하지만, 영상 초반의 Vrew 마크를 지우려면 유료 플랜이 필요해요. 이 글은 브루 공식 가격 페이지와 고객센터 공지를 2026년 10월 23일에 다시 확인해, 크레딧 계산법과 플랜 고르는 기준을 정리했어요. 목차 브루 요금 구조: 통합 크레딧이란 1크레딧으로 할 수 있는 일 무료 플랜으로 되는 것과 안 되는 것 유료 플랜 4종 비교표 월 구독·연 구독·기간 이용권, 무엇이 이득일까 상황별 추천: 학생·직장인·유튜버 크레딧이 모자랄 때: 사용 기록 지우개 업그레이드·환불 전에 알아둘 규칙 운영자 메모, 자주 묻는 질문 1. 브루 요금 구조: 통합 크레딧이란 예전 브루는 플랜마다 음성 분석 몇 분, AI 목소리 몇 자, 번역 몇 자처럼 기능별 한도를 따로 줬어요. 그런데 보이저엑스(브루 운영사)가 실제 이용 데이터를 보니 한두 가지 기능만 집중해서 쓰는 사람이 훨씬 많았다고 해요. 그래서 2026년 4월 22일 업데이트부터는 플랜마다 정해진 크레딧을 받고, 어떤 기능에 쓸지는 내가 정하는 방식으로 바뀌었어요. 플랜별 매달 지급량은 Free 200 · Light 2,000 · Standard 10,000 · Business 12,000크레딧 이에요. 공식 공지에 따르면 무료 사용자는 매달 1일에, 유료 사용자는 매월 결제일에 충전되고, 남은 크레딧은 다음 달로 넘어가지 않아요. 바뀐 뒤 무엇이 좋아졌나 공지에 나온 예시가 이해하기 쉬워요. Standard 플랜에서 AI 목소리만 쓰면 예전에는 최대 50만 자였지만, 지금은 1만 크레딧을 모두 AI 목소리에 몰아 쓸 수 있어 최대 100만 자가 돼요. 반대로 여러 기능을 골고루 쓰는 사람은 크레딧이 기능끼리 ...

에이닷 전화 vs 익시오: 통화 녹음·AI 요약, 내 통신사·폰에 맞는 앱 고르기

요약: 통화 녹음과 AI 요약을 쓰고 싶다면 먼저 내 통신사와 휴대폰 종류부터 확인해야 해요. LG유플러스 고객은 무료 앱 익시오, SKT 고객은 에이닷 전화가 가장 기능이 많고, 다른 통신사의 아이폰 사용자는 iOS에 들어 있는 기본 통화 녹음·전사 기능이 현실적인 선택이에요. 이 글은 2026-10-30에 SKT·LG유플러스·Apple 공식 페이지와 도움말을 직접 확인해 세 가지를 비교했어요. 왜 '앱 기능'보다 '내 조건'을 먼저 봐야 할까요 통화 요약 앱 비교 글을 보면 기능 목록부터 나열하는 경우가 많아요. 그런데 실제로 써 보려는 사람에게 가장 먼저 걸리는 건 기능이 아니라 이용 조건 이에요. 익시오는 LG유플러스 고객 전용이고, 에이닷 전화는 누구나 설치할 수 있지만 통역콜·baro 같은 기능은 SKT 고객에게만 열려 있어요. 아이폰은 운영체제 구조상 앱이 통화를 마음대로 녹음하기 어려워서, 통신사 앱이 안 되면 iOS 기본 기능에 기대야 해요. 그래서 이 글은 "어떤 앱이 제일 좋은가"보다 "내 통신사·내 폰에서 무엇을 쓸 수 있나" 순서로 정리했어요. 기능 차이는 그다음이에요. 한눈에 비교: 에이닷 전화 vs 익시오 vs 아이폰 기본 통화 녹음 구분 에이닷 전화 (SKT) 익시오 ixi-O (LG유플러스) 아이폰 기본 통화 녹음 (Apple) 누가 쓸 수 있나 통신사 구분 없이 앱 설치 가능. 단, 통역콜·baro·그룹통화 발신 등은 SKT 고객 전용 U+ 모바일 고객. 타 통신사, 만 14세 미만, 선불요금제 고객은 제한 (공식 사이트 FAQ상 알뜰폰 중 KB리브엠은 예외) 지원 국가·언어의 iPhone 사용자. 1명과 통화할 때만 녹음 가능 지원 OS·기기 Android 10 이상, iOS 16 이상 iOS 17 이상 iPhone 12 이상 / Android...

타입캐스트 vs 클로바더빙: 수익 나는 채널에 쓸 때 무료·유료 조건 비교

요약 타입캐스트와 클로바더빙은 둘 다 글만 입력하면 한국어 AI 목소리를 만들어 주는 서비스지만, 무료로 쓸 수 있는 범위가 꽤 다릅니다. 클로바더빙 무료는 매달 15,000자까지 쓸 수 있는 대신 수익이 생기는 채널에는 쓸 수 없고, 타입캐스트 무료는 처음에 주는 약 5분 분량의 다운로드로 끝나는 체험판에 가깝습니다. 광고 수익이 붙은 유튜브나 쇼핑몰 영상에 넣을 목소리라면 처음부터 유료 플랜 조건을 보고 고르는 편이 안전합니다. 왜 이 두 서비스를 같이 비교하나 쇼츠 내레이션, 수업 자료 더빙, 상품 소개 영상처럼 "내 목소리 대신 AI 목소리"가 필요할 때 한국 사용자들이 가장 많이 찾는 곳이 네오사피엔스의 타입캐스트 와 네이버의 클로바더빙 입니다. 기능 소개만 보면 비슷해 보이지만, 실제로 문제가 되는 건 음질보다 "이 목소리를 내 채널에 써도 되나" 입니다. 그래서 이 글은 기능 나열보다 무료 사용 범위, 출처 표기 의무, 유료 플랜 가격과 다운로드 한도를 중심으로 정리했습니다. 모든 수치는 2026년 10월 25일에 두 서비스의 공식 요금 페이지와 고객센터 도움말에서 직접 다시 확인한 내용입니다. 한눈에 보는 비교표 항목 타입캐스트 무료 타입캐스트 베이직 클로바더빙 Free 클로바더빙 Standard 월 요금 0원 9,900원 (연간 결제 시 10% 할인) 0원 19,900원 (네이버 클라우드 플랫폼에서 결제) 무료·포함 사용량 음성 생성·재생 무제한, 다운로드는 최초 3,000크레딧(약 5분) 1회 지급 월 30,000크레딧(약 60분) 월 15,000자, 다운로드 월 20회, 프로젝트 누적 5개 (매월 1일 갱신) 월 30,000자, 다운로드 월 40회, 프로젝트 누적 20개 쓸 수 있...