기본 콘텐츠로 건너뛰기

Promptfoo Tutorial: Test Your LLM Prompts Before You Ship

Summary: Promptfoo is an open-source, MIT-licensed command-line tool that runs your prompts against one or more models, checks every answer with rules you write in a YAML file, and shows the results in a local web viewer. This how-to walks through installing it, writing a first promptfooconfig.yaml, choosing assertions, reading a failing result, and wiring the eval into GitHub Actions, using steps checked against the official docs on 2026-11-09 and a small eval we actually ran on our own machine. Official screenshots and a screenshot of our local results viewer are at the end of this post.

Why test prompts at all?

Most people change a prompt, try two or three questions by hand, and ship it if the answers "look fine". That works until a small wording change quietly breaks a case you stopped checking weeks ago, such as the reply that must always include an order number, or the JSON output another script depends on. A prompt eval turns those hand checks into a repeatable list: the same inputs, the same pass/fail rules, run every time something changes.

Promptfoo does exactly that and nothing more mysterious. You describe prompts (templates with {{variables}}), providers (the models to call), and tests (variable values plus assertions). The eval command runs every combination and prints a pass/fail table; the view command opens the same results in a browser. The official GitHub page notes that Promptfoo is now part of OpenAI and remains open source and MIT licensed, and it still supports many non-OpenAI providers, including Anthropic, Google, Ollama, and custom Python or JavaScript code.

If you already trace production traffic, this fits next to our Langfuse playbook for finding costly and bad LLM answers: tracing tells you what went wrong with real users, while an eval like this stops a known problem from coming back before you deploy.

Step 1: Install Promptfoo (check Node.js first)

The installation page lists three routes: npm install -g promptfoo, running it on demand with npx promptfoo@latest, or Homebrew on Mac and Linux. The detail that trips people up is the runtime: the docs say promptfoo requires Node.js 22.22.0 or newer and recommend Node.js 24 LTS. Run node --version before anything else; if it prints something older, upgrade first (the docs show nvm install 24 and nvm use 24 as one way).

Then confirm the CLI works:

npx promptfoo@latest --version

On our machine (Node.js 24.22.1) the first npx call took about 20 seconds because it downloads the package, and it printed version 0.125.1. Later calls were fast because npx reuses its cache. If you plan to run evals often, a global install or a project dev-dependency avoids that first download each time.

Two environment variables from the official telemetry page are worth knowing on day one. Promptfoo records basic usage events by default (which command ran, which assertion types were used, the package version, and whether it runs in CI); the docs say prompts, model outputs, test cases, and API keys are not included. Set PROMPTFOO_DISABLE_TELEMETRY=1 to turn that off, and PROMPTFOO_DISABLE_UPDATE=1 to stop the update-check banner.

Step 2: Create your first config

The getting-started guide offers three ways to begin:

  • npx promptfoo@latest init --example getting-started creates a folder with a ready-made translation example and a README.
  • npx promptfoo@latest init asks you questions in the terminal and writes a config for you.
  • npx promptfoo@latest eval setup opens a browser-based setup flow if you prefer forms over YAML.

We recommend writing the file yourself once, because then you understand every line. Here is the small config we ran for this post. It pretends to check a customer-support reply template:

description: Support reply format check
prompts:
  - 'Reply to the customer about {{topic}}. Mention the order number {{order}}.'
providers:
  - echo
tests:
  - vars: {topic: a late delivery, order: "A-1042"}
    assert:
      - type: icontains
        value: 'A-1042'
      - type: not-contains
        value: 'guarantee'
  - vars: {topic: a refund, order: "B-77"}
    assert:
      - type: contains
        value: 'refund'
      - type: javascript
        value: 'output.length < 200'
  - vars: {topic: a broken charger, order: "C-9"}
    assert:
      - type: regex
        value: 'order number [A-Z]-\d+'
      - type: is-json

The echo provider simply returns the rendered prompt as the "answer". It costs nothing and needs no API key, which makes it a good way to check that your YAML, variables, and assertions behave before you spend money on real model calls. When you are ready, swap echo for real provider IDs; the docs' own example uses openai: model IDs and needs OPENAI_API_KEY set in your shell.

Step 3: Pick assertions that match what "good" means

Assertions are optional, and the docs point out that many people start by just reviewing outputs side by side. But assertions are what make an eval repeatable. The official assertion reference groups them into two families.

Deterministic checks (free and instant)

These run as plain code on the output: equals, contains, icontains (case-insensitive), regex, is-json (optionally against a JSON schema), javascript for any custom expression, plus latency and cost thresholds. Any type can be negated with a not- prefix, which is how our not-contains: guarantee rule makes sure a support reply never promises something legal hasn't approved.

Model-graded checks (cost tokens)

Types such as llm-rubric (grade the answer against a plain-English rubric) and similar (embedding similarity above a threshold) need a grading model. That is useful for tone or helpfulness, but every test now makes extra API calls. The troubleshooting page also explains a common surprise: if you see "OpenAI API key is not set" while testing a non-OpenAI model, a model-graded assertion is probably falling back to the default OpenAI grader, and you need to set the grader provider yourself under defaultTest.options.provider.

A practical rule: write deterministic checks for anything a script depends on (format, required fields, banned phrases, length), and save model-graded checks for qualities only a reader can judge.

Step 4: Run the eval and read the result

From the folder that holds the config, run:

npx promptfoo@latest eval

Our run finished in under a second with 2 passed and 1 failed. The first two tests passed all their checks. The third failed on purpose: an echoed sentence is not valid JSON, so is-json reported "Expected output to be valid JSON", while the regex check in the same test passed. The web viewer marked that row as "1 FAIL 1 PASS (0.50)", which is a useful reminder that a test passes only if all of its assertions pass (unless you group them in an assert-set with a lower threshold).

Two more details we confirmed on that run:

  • Exit code: the command exited with code 100 because one test failed. The command-line docs say this is the default and can be changed with PROMPTFOO_FAILED_TEST_EXIT_CODE. That non-zero code is what makes a CI job turn red.
  • Saving results: adding -o results.json wrote every test, its output, and the failure reason to a file you can archive or diff. For very large evals the troubleshooting page recommends --no-table and JSONL output (--output results.jsonl) to avoid running out of memory.

Then open the browser view:

npx promptfoo@latest view

It served the results at http://localhost:15500 on our machine, with filters for Failures, Passes, and Errors, plus thumbs-up/down buttons for manual rating. Our screenshot of that viewer is at the end of this post.

Step 5: Compare models or prompt versions

The real payoff comes when you list two prompts or two providers. Promptfoo runs each test against each combination and lays them out in columns, so you can see where a cheaper model starts failing your rules. The getting-started guide also shows overriding providers from the command line with -r, for example npx promptfoo@latest eval -r provider-one provider-two, which replaces the providers in the config without editing it.

If you route traffic through a gateway, our LiteLLM tutorial shows how to put OpenAI and local Ollama models behind one endpoint, which makes this kind of side-by-side comparison easier to keep consistent.

Step 6: Run it on every pull request

The official GitHub Actions guide uses promptfoo/promptfoo-action@v1 to run a before-and-after comparison whenever files in a prompts folder change, then posts a comment with a results link on the pull request. The setup in the docs comes down to:

  1. Save a workflow such as .github/workflows/prompt-eval.yml that triggers on pull_request for your prompt paths.
  2. Give the job pull-requests: write permission so it can comment.
  3. Set up Node.js 24 (the action needs Node.js 22.22.0 or newer on the runner).
  4. Add your model key as a repository secret, for example OPENAI_API_KEY under Settings > Secrets and variables > Actions.
  5. Pass github-token, prompts (a glob), and config to the action. These three inputs are listed as required.
  6. Optionally cache ~/.cache/promptfoo; the guide notes the cache stores LLM requests and outputs so repeat runs cost less.

Promptfoo vs DeepEval vs Ragas

All three are open-source evaluation tools, but they come from different directions. Promptfoo is config-first and runs from the command line; DeepEval and Ragas are Python libraries. Facts below come from each project's official pages and GitHub repositories on 2026-11-09.

Tool License / free use Paid option How you write tests Best fit
Promptfoo MIT; Community edition "Free Forever", includes all eval features and red teaming up to 10k probes/month Enterprise and On-Premise, custom pricing (contact sales) YAML config + CLI (Node.js 22.22.0+), local web viewer, GitHub Action Comparing prompts and models; CI checks without writing Python
DeepEval Apache-2.0 library; Confident AI cloud Free plan is $0 with 2 seats, 1 project, 5 test runs per week Confident AI Starter $200/month, Team $2,000/month, Enterprise custom Pytest-style Python tests; site lists 50+ metrics Python teams that want evals inside their existing test suite
Ragas Apache-2.0; installed with pip install ragas No public price list on the docs; the team offers help via office hours and contact Python library with metrics, datasets, and experiments Measuring RAG and LLM app quality with LLM-driven metrics

If your app is built in Python and already has pytest, DeepEval will feel natural. If your main question is "did this retrieval pipeline get better?", Ragas is built around that. If you want to compare prompts and models without writing code, or you work across several languages, Promptfoo's YAML approach is the lowest barrier.

Editor's note

Who it's for: developers and prompt writers who change prompts often and want a quick, repeatable pass/fail check, especially small teams that cannot afford a hosted eval platform. The Community edition covers local evals fully.

Who it's not for: non-technical users who never open a terminal, and teams that mainly need production monitoring dashboards; that is closer to tracing tools or the paid Enterprise tier.

Gotchas we verified:

  • Old Node.js is the first blocker. Check for 22.22.0 or newer before installing.
  • A failed test makes the command exit with code 100, so a script that runs eval with set -e stops there. That is intended; change it with PROMPTFOO_FAILED_TEST_EXIT_CODE only if you really mean to.
  • Model-graded assertions add API calls and can ask for an OpenAI key even when you test another provider. Set the grader explicitly.
  • Telemetry is on by default; set PROMPTFOO_DISABLE_TELEMETRY=1 if your policy requires it.

Common errors and fixes (from the official troubleshooting page): Cannot find module '@libsql/...' means npm skipped an optional platform binding, so rerun npm install --include=optional or reinstall globally; out-of-memory on big runs is fixed with --no-table, JSONL output, or a larger NODE_OPTIONS="--max-old-space-size=8192"; [object Object] in prompts means you passed an object variable, which you can access by property after setting PROMPTFOO_DISABLE_OBJECT_STRINGIFY=true; stuck evals can be capped with PROMPTFOO_EVAL_TIMEOUT_MS and PROMPTFOO_MAX_EVAL_TIME_MS. Logs are written to ~/.promptfoo/logs.

FAQ

Is Promptfoo free?

The open-source Community edition is listed as "Free Forever" on the official pricing page and includes all LLM evaluation features, all providers, and red teaming up to 10k probes per month. You still pay your model provider for any real API calls your tests make. Enterprise and On-Premise plans use custom pricing.

Do my prompts get sent to Promptfoo's servers?

Evals run on your machine and call the model providers you configure. The telemetry page says the default usage telemetry excludes prompts, model outputs, test cases, and API keys, and you can disable it with PROMPTFOO_DISABLE_TELEMETRY=1. Sharing results online is a separate, optional step.

Can I test local models instead of paid APIs?

Yes. The getting-started guide lists local models such as Ollama among supported providers, and the zero-cost echo provider is handy for checking your config before calling any model at all. If your agent code is in Python, our Pydantic AI tutorial on typed agents and structured output pairs well with is-json checks.

Related reading on lightmind24

Start with three tests for the rules you would hate to break, run them on every prompt change, and add more only when a real bug slips through.

Last verified: 2026-11-09
Sources: Promptfoo installation, Getting started, Assertions & metrics, Command line, Troubleshooting, Telemetry, Echo provider, GitHub Actions integration, Promptfoo pricing, promptfoo on GitHub, DeepEval, Confident AI pricing, DeepEval on GitHub, Ragas docs, Ragas installation, Ragas on GitHub.

Image credit: screenshots of the public Promptfoo docs (Getting started, Assertions & metrics, GitHub Actions), plus our own screenshot of the Promptfoo local results viewer from the test eval described above, all captured 2026-11-09. See the official screenshots at the end of this post.

Uploaded Image Uploaded Image Uploaded Image Uploaded Image

댓글

이 블로그의 인기 게시물

브루(Vrew) 요금제 정리: 무료 200크레딧으로 어디까지 쓸 수 있나

요약: 영상 자막 편집기 브루(Vrew)는 2026년 4월 22일부터 기능별 사용량 대신 '통합 크레딧' 하나로 요금을 매겨요. 무료 플랜은 매달 200크레딧이 들어오는데, 음성 분석(자동 자막)만 쓰면 최대 200분 분량이라 짧은 영상 몇 편은 충분하지만, 영상 초반의 Vrew 마크를 지우려면 유료 플랜이 필요해요. 이 글은 브루 공식 가격 페이지와 고객센터 공지를 2026년 10월 23일에 다시 확인해, 크레딧 계산법과 플랜 고르는 기준을 정리했어요. 목차 브루 요금 구조: 통합 크레딧이란 1크레딧으로 할 수 있는 일 무료 플랜으로 되는 것과 안 되는 것 유료 플랜 4종 비교표 월 구독·연 구독·기간 이용권, 무엇이 이득일까 상황별 추천: 학생·직장인·유튜버 크레딧이 모자랄 때: 사용 기록 지우개 업그레이드·환불 전에 알아둘 규칙 운영자 메모, 자주 묻는 질문 1. 브루 요금 구조: 통합 크레딧이란 예전 브루는 플랜마다 음성 분석 몇 분, AI 목소리 몇 자, 번역 몇 자처럼 기능별 한도를 따로 줬어요. 그런데 보이저엑스(브루 운영사)가 실제 이용 데이터를 보니 한두 가지 기능만 집중해서 쓰는 사람이 훨씬 많았다고 해요. 그래서 2026년 4월 22일 업데이트부터는 플랜마다 정해진 크레딧을 받고, 어떤 기능에 쓸지는 내가 정하는 방식으로 바뀌었어요. 플랜별 매달 지급량은 Free 200 · Light 2,000 · Standard 10,000 · Business 12,000크레딧 이에요. 공식 공지에 따르면 무료 사용자는 매달 1일에, 유료 사용자는 매월 결제일에 충전되고, 남은 크레딧은 다음 달로 넘어가지 않아요. 바뀐 뒤 무엇이 좋아졌나 공지에 나온 예시가 이해하기 쉬워요. Standard 플랜에서 AI 목소리만 쓰면 예전에는 최대 50만 자였지만, 지금은 1만 크레딧을 모두 AI 목소리에 몰아 쓸 수 있어 최대 100만 자가 돼요. 반대로 여러 기능을 골고루 쓰는 사람은 크레딧이 기능끼리 ...

에이닷 전화 vs 익시오: 통화 녹음·AI 요약, 내 통신사·폰에 맞는 앱 고르기

요약: 통화 녹음과 AI 요약을 쓰고 싶다면 먼저 내 통신사와 휴대폰 종류부터 확인해야 해요. LG유플러스 고객은 무료 앱 익시오, SKT 고객은 에이닷 전화가 가장 기능이 많고, 다른 통신사의 아이폰 사용자는 iOS에 들어 있는 기본 통화 녹음·전사 기능이 현실적인 선택이에요. 이 글은 2026-10-30에 SKT·LG유플러스·Apple 공식 페이지와 도움말을 직접 확인해 세 가지를 비교했어요. 왜 '앱 기능'보다 '내 조건'을 먼저 봐야 할까요 통화 요약 앱 비교 글을 보면 기능 목록부터 나열하는 경우가 많아요. 그런데 실제로 써 보려는 사람에게 가장 먼저 걸리는 건 기능이 아니라 이용 조건 이에요. 익시오는 LG유플러스 고객 전용이고, 에이닷 전화는 누구나 설치할 수 있지만 통역콜·baro 같은 기능은 SKT 고객에게만 열려 있어요. 아이폰은 운영체제 구조상 앱이 통화를 마음대로 녹음하기 어려워서, 통신사 앱이 안 되면 iOS 기본 기능에 기대야 해요. 그래서 이 글은 "어떤 앱이 제일 좋은가"보다 "내 통신사·내 폰에서 무엇을 쓸 수 있나" 순서로 정리했어요. 기능 차이는 그다음이에요. 한눈에 비교: 에이닷 전화 vs 익시오 vs 아이폰 기본 통화 녹음 구분 에이닷 전화 (SKT) 익시오 ixi-O (LG유플러스) 아이폰 기본 통화 녹음 (Apple) 누가 쓸 수 있나 통신사 구분 없이 앱 설치 가능. 단, 통역콜·baro·그룹통화 발신 등은 SKT 고객 전용 U+ 모바일 고객. 타 통신사, 만 14세 미만, 선불요금제 고객은 제한 (공식 사이트 FAQ상 알뜰폰 중 KB리브엠은 예외) 지원 국가·언어의 iPhone 사용자. 1명과 통화할 때만 녹음 가능 지원 OS·기기 Android 10 이상, iOS 16 이상 iOS 17 이상 iPhone 12 이상 / Android...

타입캐스트 vs 클로바더빙: 수익 나는 채널에 쓸 때 무료·유료 조건 비교

요약 타입캐스트와 클로바더빙은 둘 다 글만 입력하면 한국어 AI 목소리를 만들어 주는 서비스지만, 무료로 쓸 수 있는 범위가 꽤 다릅니다. 클로바더빙 무료는 매달 15,000자까지 쓸 수 있는 대신 수익이 생기는 채널에는 쓸 수 없고, 타입캐스트 무료는 처음에 주는 약 5분 분량의 다운로드로 끝나는 체험판에 가깝습니다. 광고 수익이 붙은 유튜브나 쇼핑몰 영상에 넣을 목소리라면 처음부터 유료 플랜 조건을 보고 고르는 편이 안전합니다. 왜 이 두 서비스를 같이 비교하나 쇼츠 내레이션, 수업 자료 더빙, 상품 소개 영상처럼 "내 목소리 대신 AI 목소리"가 필요할 때 한국 사용자들이 가장 많이 찾는 곳이 네오사피엔스의 타입캐스트 와 네이버의 클로바더빙 입니다. 기능 소개만 보면 비슷해 보이지만, 실제로 문제가 되는 건 음질보다 "이 목소리를 내 채널에 써도 되나" 입니다. 그래서 이 글은 기능 나열보다 무료 사용 범위, 출처 표기 의무, 유료 플랜 가격과 다운로드 한도를 중심으로 정리했습니다. 모든 수치는 2026년 10월 25일에 두 서비스의 공식 요금 페이지와 고객센터 도움말에서 직접 다시 확인한 내용입니다. 한눈에 보는 비교표 항목 타입캐스트 무료 타입캐스트 베이직 클로바더빙 Free 클로바더빙 Standard 월 요금 0원 9,900원 (연간 결제 시 10% 할인) 0원 19,900원 (네이버 클라우드 플랫폼에서 결제) 무료·포함 사용량 음성 생성·재생 무제한, 다운로드는 최초 3,000크레딧(약 5분) 1회 지급 월 30,000크레딧(약 60분) 월 15,000자, 다운로드 월 20회, 프로젝트 누적 5개 (매월 1일 갱신) 월 30,000자, 다운로드 월 40회, 프로젝트 누적 20개 쓸 수 있...