Braintrust
Developer ToolsRun LLM evaluations and record experiment results with Braintrust, logging production traces alongside them. Agents score model output against a dataset before a prompt change reaches customers.
What This Integration Enables
A prompt change is a code change. It alters what your software says to customers, and it ships with none of the protection a code change gets: no diff review that means anything, no test suite, no red build. Braintrust closes that gap by making the scored run a first-class object. FlowRunner agents can create a project, load golden test cases into a dataset, launch a hosted evaluation over that dataset, and read back the aggregate with a comparison against the previous run. The point is not that a number exists. The point is that a release now has something to fail against.
The connector reaches all seven surfaces of the Braintrust API: projects, experiments, datasets, production logs, evaluations, prompts and functions. That range matters because evaluation only stays honest if the dataset keeps absorbing reality. An agent can pull live traffic out of a project log, filter it to the calls a user rated badly, and propose those inputs as new dataset rows. It can push traces from model calls made anywhere else in a flow into the same project log, so the observability record and the evaluation record share a home. And it can fetch a prompt or invoke a stored function at runtime, which means the definition your team approved in Braintrust is the definition that runs, rather than a copy that drifted. Every one of these operations is built and verified against the vendor's official API.
Without FlowRunner
With FlowRunner
Use Case Scenarios
Turning bad answers into permanent test cases
The support triage assistant has been live for a week. Its calls were instrumented with Insert Project Log Events, and end-user thumbs-down ratings were written back with Log Project Feedback, so the project log holds both what the model said and what people thought of it. A weekly flow calls Fetch Project Log Events, keeps only the negatively rated calls, and checks each input against Fetch Dataset Events so the same case is not added twice. The remaining candidates go to the eval owner, who supplies the answer the assistant should have given. Insert Dataset Events writes the approved rows. The next release is measured against a set that grew from the assistant's actual failures rather than from what someone imagined it might get wrong.
A release gate that names the thing that broke
An engineer publishes a new version of the extraction prompt and merges it in GitHub. The merge starts a flow that calls Run Eval, which executes the task over the golden dataset, scores every result with the configured scorers, and records the run as a new experiment. Summarize Experiment then returns the per-scorer averages together with the change against the previous release. The flow posts that comparison to the engineering channel in Slack with the failing scorer called out by name. The conversation stops being "does this feel better" and becomes "faithfulness dropped and accuracy held, is that trade acceptable here".
Human ratings and machine scores in one place
A reviewer is asked to judge a sample of the assistant's answers. The flow fetches the sample with Fetch Experiment Events, presents each one for a verdict, and writes the verdict back with Log Experiment Feedback as a named numeric score plus a written comment. The reviewer's judgement now sits on the same event as the automated scorer's judgement. When the two disagree, that disagreement is the interesting artifact: it usually means the scorer is measuring something other than what the team actually cares about, and the fix is to the scorer rather than to the model.
Human-in-Loop Highlight
The gate is not on the evaluation. It is on the dataset write. Insert Dataset Events treats an existing row ID as an update rather than an addition, so writing a golden row is both an assertion about what correct looks like and a permanent change to the standard every future run is graded against. An agent that samples a bad answer out of the project log and files the model's own output as the expected value has quietly promoted a mistake to ground truth, and every experiment after that will score it as a pass. So the agent stops. It posts each candidate row to the eval owner with the original input, the output the model produced, and the user feedback that flagged it, and asks: "This input is not yet in the golden set. Should the expected output be the model's answer as written, your correction, or should this case be dropped as unrepresentative?" Nothing is written until the owner answers. This is human-in-the-loop applied at the one point in an evaluation pipeline where an agent acting alone corrupts the measurement instead of failing loudly.
Agent Capabilities
22 actionsProjects
3- List Projects Retrieves the Braintrust projects visible to the API key, newest first. Projects are the top-level container for experiments, datasets, logs, prompts and scorers, so this is normally the first call in a flow that needs a project ID. Supports name filtering, organization scoping and cursor pagination.
- Create Project Creates a new Braintrust project, or returns the existing one if a project with the same name already exists in the organization. Use it to provision a workspace for a new application or environment from a flow rather than through the UI.
- Get Project Retrieves a single project by ID, including its organization, settings and creation time. Use it to confirm a project still exists, or to read its settings before writing experiments or logs into it.
Experiments
6- List Experiments Retrieves experiments, optionally narrowed to a project or a specific experiment name. Each experiment is one evaluation run over a dataset, so this is how you find the run whose scores you want to read or compare.
- Create Experiment Creates an experiment inside a project to hold the results of an evaluation run. Optionally links it to a dataset, sets a base experiment for regression comparison, and attaches metadata and tags you can later slice results by.
- Insert Experiment Events Writes one or more evaluation events into an experiment. Each event carries the input, the output your system produced, the expected value, computed scores and any metadata, and appears in Braintrust as a row of the run. Use it to record results from a task executed elsewhere in a flow rather than inside a Braintrust-hosted eval.
- Fetch Experiment Events Retrieves the recorded events of an experiment, newest first, with their inputs, outputs, expected values and scores. A version can be supplied to read the experiment as it stood at an earlier point in time.
- Summarize Experiment Returns aggregate metrics for an experiment: average scores per scorer, token usage, duration and, when a comparison experiment is supplied, the improvement or regression against it. This is the call a release gate reads.
- Log Experiment Feedback Attaches human feedback to specific events in an experiment: numeric scores, a written comment, or extra metadata. Use it to fold reviewer judgements back into an evaluation run so human and automated scores sit side by side.
Datasets
5- List Datasets Retrieves datasets, optionally narrowed to a project or a specific dataset name. Datasets hold the golden inputs and expected outputs your evaluations run against, so this is how you locate the dataset ID to insert rows into or evaluate.
- Create Dataset Creates a dataset inside a project to hold evaluation test cases. A dataset starts empty; add rows with Insert Dataset Events and then reference it from Create Experiment or Run Eval.
- Insert Dataset Events Adds rows to a dataset. Each row typically carries an input and the expected output, plus optional metadata and tags. Supplying an existing row ID updates that row instead of adding a new one, which makes this safe to run repeatedly when syncing test cases from another system.
- Fetch Dataset Events Retrieves the rows of a dataset, newest first, with cursor pagination. Supply a version to read the dataset as it stood at an earlier point, which is how an old evaluation is reproduced exactly.
- Summarize Dataset Returns summary statistics for a dataset, including how many rows it holds and which experiments have been run against it. Use it to confirm a dataset is populated before launching an evaluation.
Production Logs
3- Insert Project Log Events Writes trace events into a project's production log, Braintrust's observability store for live traffic as opposed to evaluation runs. Each event records an input, output, metrics and metadata, and events can be linked into a trace via span parent references. Use it to instrument model calls made elsewhere in a flow.
- Fetch Project Log Events Retrieves events from a project's production log, newest first, with cursor pagination. Use it to pull recent live traffic into a flow, for example to sample real requests into an evaluation dataset or to review what an agent actually did.
- Log Project Feedback Attaches feedback to events in a project's production log: numeric scores, a written comment or extra metadata. This is how end-user ratings or reviewer judgements on live traffic get recorded against the original trace.
Evaluations
1- Run Eval Launches a Braintrust-hosted evaluation: runs the given task function over a dataset, scores every result with the supplied scorers, and records everything as a new experiment. Supports multiple trials per input to measure non-determinism, a base experiment for regression comparison, and concurrency and timeout limits. Returns the resulting experiment so its summary can gate a release.
Prompts and Functions
4- List Prompts Retrieves prompts stored in Braintrust, optionally narrowed to a project, name or slug. Prompts are versioned, and an environment slug or version ID can be supplied to resolve the exact revision deployed to a given environment.
- Get Prompt Retrieves a single prompt by ID, returning its messages, model configuration and declared parameters. Use it to fetch the approved prompt at runtime so prompt changes ship from Braintrust without redeploying the flow.
- List Functions Retrieves the functions defined in Braintrust: prompts, code scorers and tools that can be invoked directly or used as tasks and scorers in an evaluation. Filter by project, name or slug, and resolve a specific version or deployed environment.
- Invoke Function Executes a Braintrust function, a stored prompt, code scorer or tool, with the supplied input and returns its result. The call is traced automatically, so metadata and tags passed here appear on the resulting span. Use it to run a centrally managed prompt or scorer without duplicating its definition in the flow.
Frequently Asked Questions
What can FlowRunner do with Braintrust?
FlowRunner agents can run List Projects, Create Project, and Get Project in Braintrust, plus 19 more actions.
Does connecting Braintrust to FlowRunner require OAuth?
No. Braintrust connects to FlowRunner with an API key, no OAuth flow required.
Can Braintrust trigger a FlowRunner workflow automatically?
Braintrust doesn't currently expose triggers in FlowRunner. It connects as an action step inside workflows started by another trigger.
Start building with Braintrust
$100 in credits. No card required. Connect in minutes.