기본 콘텐츠로 건너뛰기

Langfuse Playbook: Find Costly and Bad LLM Answers

Summary: Langfuse is an open-source tool that records every LLM call in your app, works out what each one cost, and lets you score answers with user feedback or an LLM judge. This playbook takes one concrete case, a support chatbot that is getting expensive and sometimes answers badly, and walks through the setup that finds both problems in about a week of real traffic. The free Hobby cloud plan (50k units a month) or a free self-hosted install is enough to follow along.

The scenario: a chatbot nobody can explain

Picture a small product team that shipped a help-center chatbot a few months ago. The model bill keeps creeping up, a few customers have complained about wrong answers, and when someone asks "which conversations cost the most?" or "how often is it wrong?", nobody can answer. The provider dashboard shows total tokens, but not which feature, user or prompt produced them.

That gap is what LLM observability tools are for. The plan below uses Langfuse to answer four questions, in this order:

  1. What does one conversation actually cost, and which ones are outliers?
  2. Which answers do real users dislike?
  3. Can a judge model flag weak answers automatically, so you do not depend on the few users who click a button?
  4. How do you keep the Langfuse bill itself under control as traffic grows?

Step 1: Choose cloud or self-hosted

The instrumentation code is the same for both, so this is mainly a question of who runs the servers.

  • Langfuse Cloud: sign up at cloud.langfuse.com, create a project, and create API credentials in the project settings. You get a public key (pk-lf-...) and a secret key (sk-lf-...). Data regions are EU (https://cloud.langfuse.com), US (https://us.cloud.langfuse.com) and Japan (https://jp.cloud.langfuse.com), plus a HIPAA region.
  • Self-hosted: the core is MIT-licensed with unlimited usage. For a trial, clone the langfuse/langfuse repository, update the secrets in docker-compose.yml, and run docker compose up. The docs say the web container logs "Ready" after about two to three minutes, and the UI is then at http://localhost:3000. There is no default admin account: click Sign up to create the first user. For a VM, the docs recommend at least 4 cores, 16 GiB of memory and around 100 GiB of storage, and they warn that the Compose setup has no high availability or backups, so use Kubernetes for serious production.

If your chat logs contain personal data, decide on the region (or self-hosting) before you send a single trace. Moving later is more work than choosing now.

Step 2: Trace every chatbot call

If the bot uses the OpenAI Python SDK, the change is one import. Install the SDK with pip install langfuse, set three environment variables, and swap the import:

LANGFUSE_SECRET_KEY="sk-lf-..."
LANGFUSE_PUBLIC_KEY="pk-lf-..."
LANGFUSE_BASE_URL="https://cloud.langfuse.com"

# before: import openai
from langfuse.openai import openai

The wrapper behaves like the normal client and records the prompt, model and output for each call. For JavaScript and TypeScript, the official route is @langfuse/openai plus @langfuse/otel: start an OpenTelemetry NodeSDK with a LangfuseSpanProcessor before any traced code runs, then wrap the client with observeOpenAI(new OpenAI()). LangChain users add a CallbackHandler instead, and the Vercel AI SDK has its own integration.

Next, attach the context you will want to filter by later. In Python, wrap the request handler in propagate_attributes() so every nested call inherits the same values:

from langfuse import observe, propagate_attributes

@observe()
def answer(user_id, chat_id, question):
    with propagate_attributes(user_id=user_id,
                              session_id=chat_id,
                              tags=["support-bot"]):
        return run_bot(question)

The session_id groups every turn of one chat into a single replayable conversation (it must be a US-ASCII string under 200 characters). The user_id lets you rank users by cost, and the tag lets you separate the support bot from any other LLM feature in the same project. These three fields are what turn a pile of logs into answers.

Step 3: Find the expensive conversations

You usually do not need to configure anything for cost. Langfuse ships with prices for popular OpenAI, Anthropic and Google models, and most integrations capture token usage automatically. For each generation it stores usage and cost broken down by type, such as input, output and cached tokens. If you send cost yourself, the ingested value wins over the inferred one.

After a few days of traffic, open the cost dashboard and look at cost by model, cost over time, and the top users and use cases by cost. In a support bot, the outliers are usually long sessions where the full history is re-sent on every turn, or a retrieval step that stuffs too many documents into the prompt. Open one of those sessions and you can see each turn's tokens side by side.

Two cost gotchas to check early:

  • A generation shows no cost. The model name did not match any model definition, which is common with fine-tuned, self-hosted or very new models. Click the model badge on the generation, or go to Project Settings > Models, and add a definition with a regex match pattern and prices. It applies to new traces only, not old ones.
  • Long-context pricing. Some models charge more above a context threshold. Langfuse supports pricing tiers with conditions on usage, model parameters or metadata, so a custom definition can mirror that.

Step 4: Capture what users think

Cost is half the story; the other half is whether answers helped. In Langfuse, feedback is stored as a score linked to a trace. The documented pattern for a web chat is:

  1. Have the backend return the trace ID with each answer.
  2. Add thumbs-up and thumbs-down buttons that send a score (for example user-feedback, 1 or 0, type BOOLEAN) through the Langfuse browser SDK. Use a dedicated key that is only allowed to ingest scores, since it lives in the browser.
  3. In Langfuse, filter traces by user-feedback < 1 to read the disliked answers.

You can also score from the server. A good implicit signal for support is ticket outcome: call langfuse.create_score() with 1 when the ticket closes without a human, and 0 when it is escalated. That catches unhappy users who never click anything.

Step 5: Let a judge model score answers automatically

Explicit feedback is sparse, so the next step is an LLM-as-a-Judge evaluator that scores live answers against a rubric.

  1. Set up an LLM Connection (the provider key the judge will use).
  2. Go to Evaluators, click New evaluator, and start from a Langfuse template or a blank LLM-as-a-Judge prompt.
  3. Write the rubric with variables such as {{input}} and {{output}}, and pick a score type: numeric (for example helpfulness from 0 to 1), categorical labels, or boolean.
  4. Map each variable to the observation's input, output, metadata or tool calls.
  5. Test it on sample observations in the right-hand panel, read the score and reasoning, and adjust until it agrees with you.
  6. Save, then create a rule that runs it on incoming observations.

Two details from the docs matter for a chatbot. First, observation-level evaluators are the recommended target for live data, and trace-level ones are deprecated. Second, an observation-level evaluator does not see sibling or child observations. If the judge needs the whole question and final answer, target a logical root observation that records both, using the Is Root Observation filter in the rule. Before trusting the judge, compare its scores with a small set of answers you labeled by hand; Langfuse's Score Analytics can show that agreement.

Step 6: Keep the Langfuse bill predictable

Cloud pricing is based on units: traces plus observations plus scores. Units created by Langfuse features count too, including LLM-as-a-Judge scores, annotation queues and experiments. As a rough worked example, a chat turn recorded as one trace, three observations and one judge score uses five units, so the Hobby plan's 50k monthly units would cover about 10,000 such turns. Your own count depends on how many steps your bot logs; the built-in Langfuse Usage Management dashboard shows the real totals.

Three levers keep it in check:

  • Sampling: set LANGFUSE_SAMPLE_RATE (or sample_rate) between 0 and 1. Sampling is per trace, so a dropped trace takes its observations and scores with it. In JS/TS, use an OpenTelemetry sampler such as TraceIdRatioBasedSampler(0.2).
  • Judge only what matters: filter the evaluator rule to the final answer observation and the support-bot tag instead of every step.
  • Spend alerts: on Core and above, set email alerts in the organization's Billing tab. They track your Langfuse bill (checked every 60 to 90 minutes), not your model spend.

For privacy, the Python SDK's mask_otel_spans hook can delete or rewrite span attributes before data leaves your app, which is the place to strip emails or order numbers from chat content.

Langfuse plans and two alternatives

Option Free tier First paid plan Worth knowing
Langfuse Cloud Hobby: 50k units/month, 30 days of data access, 2 users Core $29/month: 100k units, then $8 per 100k; 90 days of data access; unlimited users Pro $199/month for 3 years of access and retention controls; core features can also be self-hosted free under MIT
LangSmith (LangChain) Developer: $0, 1 seat, up to 5k base traces/month Plus $39/seat/month: up to 10k base traces/month, then pay as you go Self-hosted and hybrid deployment are listed under Enterprise (custom pricing)
Helicone Hobby: 10,000 requests, 1 GB storage, 1 seat, 7-day retention Pro $79/month: unlimited seats, alerts and reports, usage-based pricing beyond the free allowance Includes gateway features such as caching, rate limits and fallbacks

Prices as listed on each vendor's official pricing page on 2026-11-02. Units, traces and requests are counted differently, so compare them against your own traffic rather than side by side.

Editor's note

Good fit: teams with an LLM feature already in production who need per-user cost and a quality signal, and anyone who wants the option to self-host the whole stack for free.

Not a fit: a single prompt you run by hand a few times a week. The provider's usage page is enough there. And if you mainly want a proxy with caching and fallbacks, a gateway-first tool may suit you better.

Common errors and fixes from the official troubleshooting page:

  • No traces appear. Check that the three LANGFUSE_* variables are set, that tracing_enabled is true and sample_rate is not 0. In scripts, serverless functions and batch jobs, call langfuse.flush() or langfuse.shutdown() before exit, because data is sent in the background. Use langfuse.auth_check() during setup (not in production) to test the keys.
  • OpenAI calls are missing. The wrapper or callback handler must be created before the API calls run.
  • Steps show up at the top level instead of nested. Use context managers, call .end() on manual observations, and make sure the parent observation is exported. Self-hosted servers need version 3.63.0 or later for the OpenTelemetry-based SDKs.
  • Next.js traces vanish with @vercel/otel. Upgrade to v2 or later, or register the span processor through a manual NodeSDK.

FAQ

Is Langfuse really free?

Two ways. The Cloud Hobby plan is free with no credit card, limited to 50k units a month, 30 days of data access and 2 users. The self-hosted open-source version is free under the MIT license with unlimited usage, but you pay for and maintain the servers.

Do I have to use LangChain?

No. Langfuse has drop-in wrappers for the OpenAI SDK in Python and JS/TS, a Vercel AI SDK integration, OpenTelemetry support for other languages, and proxy-based logging through LiteLLM. LangChain is just one of many integrations.

Do LLM-as-a-Judge evaluations cost money?

Yes, in two places. The judge model runs on your own LLM Connection, so the provider bills you for those tokens, and each score it writes counts as a Langfuse unit on Cloud. Filtering the rule to final answers keeps both small.

Related on lightmind24: if you have not built the bot yet, the Dify playbook for a help-docs chatbot is a quick way to start. Teams running an internal chat UI can pair this with the Open WebUI playbook, and the OpenRouter free models guide helps once your cost data shows which calls could move to a cheaper model.

Instrument first, read a week of real conversations, and only then decide which prompt or model to change.

Images: official Langfuse documentation images and a screenshot of the langfuse.com pricing page captured by the author; see the official screenshots at the end of this post.

Last verified: 2026-11-02
Sources:
Langfuse Cloud pricing
Langfuse self-hosted pricing
Langfuse docs: Get Started with Tracing
Langfuse docs: Sessions
Langfuse docs: Token and Cost Tracking
Langfuse docs: User Feedback
Langfuse docs: LLM-as-a-Judge
Langfuse docs: Billable Units
Langfuse docs: Sampling
Langfuse docs: Spend Alerts
Langfuse docs: Masking
Langfuse docs: SDK Troubleshooting
Langfuse docs: Docker Compose deployment
LangSmith pricing
Helicone pricing

Uploaded Image Uploaded Image Uploaded Image Uploaded Image

댓글

이 블로그의 인기 게시물

브루(Vrew) 요금제 정리: 무료 200크레딧으로 어디까지 쓸 수 있나

요약: 영상 자막 편집기 브루(Vrew)는 2026년 4월 22일부터 기능별 사용량 대신 '통합 크레딧' 하나로 요금을 매겨요. 무료 플랜은 매달 200크레딧이 들어오는데, 음성 분석(자동 자막)만 쓰면 최대 200분 분량이라 짧은 영상 몇 편은 충분하지만, 영상 초반의 Vrew 마크를 지우려면 유료 플랜이 필요해요. 이 글은 브루 공식 가격 페이지와 고객센터 공지를 2026년 10월 23일에 다시 확인해, 크레딧 계산법과 플랜 고르는 기준을 정리했어요. 목차 브루 요금 구조: 통합 크레딧이란 1크레딧으로 할 수 있는 일 무료 플랜으로 되는 것과 안 되는 것 유료 플랜 4종 비교표 월 구독·연 구독·기간 이용권, 무엇이 이득일까 상황별 추천: 학생·직장인·유튜버 크레딧이 모자랄 때: 사용 기록 지우개 업그레이드·환불 전에 알아둘 규칙 운영자 메모, 자주 묻는 질문 1. 브루 요금 구조: 통합 크레딧이란 예전 브루는 플랜마다 음성 분석 몇 분, AI 목소리 몇 자, 번역 몇 자처럼 기능별 한도를 따로 줬어요. 그런데 보이저엑스(브루 운영사)가 실제 이용 데이터를 보니 한두 가지 기능만 집중해서 쓰는 사람이 훨씬 많았다고 해요. 그래서 2026년 4월 22일 업데이트부터는 플랜마다 정해진 크레딧을 받고, 어떤 기능에 쓸지는 내가 정하는 방식으로 바뀌었어요. 플랜별 매달 지급량은 Free 200 · Light 2,000 · Standard 10,000 · Business 12,000크레딧 이에요. 공식 공지에 따르면 무료 사용자는 매달 1일에, 유료 사용자는 매월 결제일에 충전되고, 남은 크레딧은 다음 달로 넘어가지 않아요. 바뀐 뒤 무엇이 좋아졌나 공지에 나온 예시가 이해하기 쉬워요. Standard 플랜에서 AI 목소리만 쓰면 예전에는 최대 50만 자였지만, 지금은 1만 크레딧을 모두 AI 목소리에 몰아 쓸 수 있어 최대 100만 자가 돼요. 반대로 여러 기능을 골고루 쓰는 사람은 크레딧이 기능끼리 ...

에이닷 전화 vs 익시오: 통화 녹음·AI 요약, 내 통신사·폰에 맞는 앱 고르기

요약: 통화 녹음과 AI 요약을 쓰고 싶다면 먼저 내 통신사와 휴대폰 종류부터 확인해야 해요. LG유플러스 고객은 무료 앱 익시오, SKT 고객은 에이닷 전화가 가장 기능이 많고, 다른 통신사의 아이폰 사용자는 iOS에 들어 있는 기본 통화 녹음·전사 기능이 현실적인 선택이에요. 이 글은 2026-10-30에 SKT·LG유플러스·Apple 공식 페이지와 도움말을 직접 확인해 세 가지를 비교했어요. 왜 '앱 기능'보다 '내 조건'을 먼저 봐야 할까요 통화 요약 앱 비교 글을 보면 기능 목록부터 나열하는 경우가 많아요. 그런데 실제로 써 보려는 사람에게 가장 먼저 걸리는 건 기능이 아니라 이용 조건 이에요. 익시오는 LG유플러스 고객 전용이고, 에이닷 전화는 누구나 설치할 수 있지만 통역콜·baro 같은 기능은 SKT 고객에게만 열려 있어요. 아이폰은 운영체제 구조상 앱이 통화를 마음대로 녹음하기 어려워서, 통신사 앱이 안 되면 iOS 기본 기능에 기대야 해요. 그래서 이 글은 "어떤 앱이 제일 좋은가"보다 "내 통신사·내 폰에서 무엇을 쓸 수 있나" 순서로 정리했어요. 기능 차이는 그다음이에요. 한눈에 비교: 에이닷 전화 vs 익시오 vs 아이폰 기본 통화 녹음 구분 에이닷 전화 (SKT) 익시오 ixi-O (LG유플러스) 아이폰 기본 통화 녹음 (Apple) 누가 쓸 수 있나 통신사 구분 없이 앱 설치 가능. 단, 통역콜·baro·그룹통화 발신 등은 SKT 고객 전용 U+ 모바일 고객. 타 통신사, 만 14세 미만, 선불요금제 고객은 제한 (공식 사이트 FAQ상 알뜰폰 중 KB리브엠은 예외) 지원 국가·언어의 iPhone 사용자. 1명과 통화할 때만 녹음 가능 지원 OS·기기 Android 10 이상, iOS 16 이상 iOS 17 이상 iPhone 12 이상 / Android...

타입캐스트 vs 클로바더빙: 수익 나는 채널에 쓸 때 무료·유료 조건 비교

요약 타입캐스트와 클로바더빙은 둘 다 글만 입력하면 한국어 AI 목소리를 만들어 주는 서비스지만, 무료로 쓸 수 있는 범위가 꽤 다릅니다. 클로바더빙 무료는 매달 15,000자까지 쓸 수 있는 대신 수익이 생기는 채널에는 쓸 수 없고, 타입캐스트 무료는 처음에 주는 약 5분 분량의 다운로드로 끝나는 체험판에 가깝습니다. 광고 수익이 붙은 유튜브나 쇼핑몰 영상에 넣을 목소리라면 처음부터 유료 플랜 조건을 보고 고르는 편이 안전합니다. 왜 이 두 서비스를 같이 비교하나 쇼츠 내레이션, 수업 자료 더빙, 상품 소개 영상처럼 "내 목소리 대신 AI 목소리"가 필요할 때 한국 사용자들이 가장 많이 찾는 곳이 네오사피엔스의 타입캐스트 와 네이버의 클로바더빙 입니다. 기능 소개만 보면 비슷해 보이지만, 실제로 문제가 되는 건 음질보다 "이 목소리를 내 채널에 써도 되나" 입니다. 그래서 이 글은 기능 나열보다 무료 사용 범위, 출처 표기 의무, 유료 플랜 가격과 다운로드 한도를 중심으로 정리했습니다. 모든 수치는 2026년 10월 25일에 두 서비스의 공식 요금 페이지와 고객센터 도움말에서 직접 다시 확인한 내용입니다. 한눈에 보는 비교표 항목 타입캐스트 무료 타입캐스트 베이직 클로바더빙 Free 클로바더빙 Standard 월 요금 0원 9,900원 (연간 결제 시 10% 할인) 0원 19,900원 (네이버 클라우드 플랫폼에서 결제) 무료·포함 사용량 음성 생성·재생 무제한, 다운로드는 최초 3,000크레딧(약 5분) 1회 지급 월 30,000크레딧(약 60분) 월 15,000자, 다운로드 월 20회, 프로젝트 누적 5개 (매월 1일 갱신) 월 30,000자, 다운로드 월 40회, 프로젝트 누적 20개 쓸 수 있...