PromptLayer
Developer ToolsManage prompts, observe model calls, and run evaluations with PromptLayer, fetching versioned prompts by release label. Agents run the prompt version that was approved rather than one pasted into code.
What This Integration Enables
PromptLayer treats a prompt as a versioned artifact with a release label pointing at it, and that single design choice is what this connector is for. A flow does not carry prompt text. It calls Get Prompt Template with a label such as prod, receives the blueprint and model settings that were approved, and runs those. When someone publishes a better version, they publish it to the registry; when someone decides it is ready, they move the label. Nothing in the flow is redeployed and nothing is retyped. The prompt that runs in production is the one that was approved, which is a lower bar than most teams clear today.
The rest of the surface exists to make that promotion decision defensible. Log Request records a completed model call with its provider, tokens, price, latency and the template version that produced it, and returns an ID that Track Score, Track Metadata and Track Group then hang further evidence on. Search Request Logs and Get Request Analytics turn that record into answers about quality and spend, including a breakdown by any metadata key you attribute costs with. Run Evaluation Pipeline and Get Evaluation Score put a number on a candidate version before it is promoted. If you also run Braintrust, the two are complements rather than duplicates: Braintrust centres on the experiment and the dataset it runs against, while PromptLayer centres on the prompt version and the label that decides which one is live. All of these operations are built and verified against the vendor's official API.
Without FlowRunner
With FlowRunner
Use Case Scenarios
The flow that owns no prompt text
A classification flow starts by calling Get Prompt Template with the template name and the prod label, and receives the rendered blueprint along with the model settings recorded on that version. It calls Anthropic with those settings, then calls Log Request with the input, the output, the token counts and a reference back to the template version. Track Group stitches that call to the other steps of the same run so the whole chain is reviewable as one unit. A prompt engineer can now change behaviour by publishing a version and moving a label. Nobody edits a flow, and nobody has to guess which copy of the prompt is authoritative, because there is only one.
Attributing model spend to the customers it was spent on
Every logged request carries metadata for the customer and the feature that made the call. On a schedule, a flow calls Get Request Analytics with a date window and a metadata key to break down by, and receives volume, tokens, cost and latency per value of that key. It appends the result to a sheet in Google Sheets that finance already reads, and posts the top movers to Slack. The question "which accounts are expensive to serve" stops being an engineering archaeology project and becomes a row in a report.
Running a PromptLayer agent and collecting a human verdict
A flow calls Run Agent with input variables, pinned to a release label so behaviour stays stable, and gets an execution ID back immediately. It polls Get Agent Execution Results until the run finishes, then presents the output to a reviewer in Slack for a rating. When the rating comes back, Track Score writes it against the logged request as a named score, so a human judgement sits beside any automated grader on the same record. Over time those named scores are what a promotion argument is actually built from.
Human-in-Loop Highlight
Create Prompt Label is the irreversible moment, and it does not look like one. It is a small call that attaches a label such as prod to a version, and from that instant every application and flow fetching the template by that label receives the new version. There is no staged rollout, no percentage, and no per-consumer opt-in; the change is complete before anyone observes it. Delete Prompt Label is worse in its way, because consumers fetching a deleted label stop resolving to a pinned version at all. So the agent runs the evaluation, reads the score, and then stops. It sends the reviewer the commit message from the candidate version and the aggregate score beside the score of the version currently live. It adds the answer from List Prompt Template Labels, showing exactly which version prod resolves to right now. Then it asks: "Promoting version 14 to prod will switch every flow reading this label on the next call. Move the label, or hold at version 11?" The digital andon cord gets pulled at the label, not at the publish, because publishing a version harms nothing and pointing production at it changes everything. Human-in-the-loop is the default here rather than an option you configure.
Agent Capabilities
20 actionsPrompt Registry
3- List Prompt Templates Retrieves a paginated list of prompt templates from the PromptLayer registry for the workspace tied to the API key. Supports filtering by name, release label, tags, creator email, creation and update windows, and archive status, so it can be used both to browse the registry and to audit recent prompt changes.
- Get Prompt Template Fetches a single prompt template by name or ID and returns its provider-agnostic prompt blueprint, model settings and metadata. Supply input variables to have PromptLayer render the template server-side, or pin the result to a specific version number or release label so production flows always fetch the approved prompt.
- Publish Prompt Template Creates a new version of a prompt template in the registry, or creates the template outright if the name is new. Accepts a full prompt blueprint, an optional commit message describing the change, tags, version metadata such as the target model, and release labels to move onto the new version in the same call.
Release Labels
3- List Prompt Template Labels Retrieves every release label attached to a prompt template, along with the version each label currently points at. Use it to confirm which version is live in production before promoting a new one.
- Create Prompt Label Attaches a new release label to a specific version of a prompt template, which is how a version is promoted to an environment such as prod or staging. Applications that fetch the template by label immediately begin receiving the labelled version.
- Delete Prompt Label Permanently removes a release label from a prompt template. Any application fetching that template by the deleted label will stop resolving to a pinned version, so retire labels only once dependent flows have been repointed.
Request Logging
4- Log Request Records a completed model call in PromptLayer so it appears in the dashboard alongside SDK-instrumented traffic. Captures the provider, model, prompt input and model output, token counts, price, latency window, tags and custom metadata, and can link the call back to the prompt template and version that produced it. Returns the request ID needed by Track Score, Track Metadata and Track Group.
- Track Score Attaches a quality score between 0 and 100 to a previously logged request. Named scores let several evaluation signals coexist on the same request, for example a human rating alongside an automated grader, and they become filterable and chartable in the dashboard.
- Track Metadata Adds or updates custom key-value metadata on a previously logged request, letting you enrich a log after the fact with information that was not available at call time such as a resolved user ID, a downstream outcome or a support ticket reference. Metadata is filterable and groupable in search and analytics.
- Track Group Associates a logged request with a PromptLayer group, which is how multi-step chains and agent runs are stitched together into a single reviewable unit. Call it once per request in the chain using the same group ID so the whole interaction can be inspected end to end.
Log Search and Analytics
3- Search Request Logs Searches the workspace's request logs using free-text matching over prompt input and model output, structured filters combined with AND/OR logic, sorting and pagination. Returns per-request summaries including model, token counts, price, latency, tags, metadata and any tracked scores.
- Get Request Log Retrieves the full record of a single logged request, including the replayable prompt blueprint, provider and model, token counts, price, latency and the associated trace ID. Use it to inspect an individual call surfaced by Search Request Logs.
- Get Request Analytics Returns aggregated analytics across the workspace's request logs for the supplied filters, covering volume, token usage, cost and latency. Optionally breaks cost and request counts down by the values of a chosen metadata key, which is the standard way to attribute model spend to a customer, feature or environment.
Agents
3- List Agents Retrieves a paginated list of the workspace's PromptLayer agents, including the latest version number and any release labels attached to each. Supports filtering by creator and creation or update windows, and sorting by name or timestamp.
- Run Agent Starts an execution of a PromptLayer agent with the supplied input variables and returns the execution ID immediately; the run itself continues asynchronously. Pin the run to a specific version number or release label to keep production behaviour stable.
- Get Agent Execution Results Retrieves the results of an agent execution started by Run Agent. By default only the final output node is returned; enable Return All Outputs to receive the result of every node, which is useful when debugging a multi-step agent. Poll this operation until the execution completes.
Evaluation
4- List Datasets Retrieves a paginated list of evaluation datasets in the workspace, including each dataset's version number and column names. Supports filtering by dataset group, prompt, prompt version, report, name and archive status, so it can locate the exact dataset version an evaluation should run against.
- List Evaluations Retrieves the workspace's evaluation pipelines and, optionally, the batch runs nested beneath each one with their status and report data. Use it to discover the pipeline ID needed by Run Evaluation Pipeline, or to monitor whether recent runs have completed.
- Run Evaluation Pipeline Executes an existing evaluation pipeline over a dataset and creates a named batch report. If no dataset is supplied the pipeline's default dataset is used, and dynamic datasets can be refreshed from their filter parameters before the run. Returns the new report ID, which Get Evaluation Score reads once the run completes.
- Get Evaluation Score Retrieves the aggregate score for an evaluation report, together with the scoring strategy used and the per-column detail behind it. Use it after Run Evaluation Pipeline to gate a prompt release on a quality threshold.
Frequently Asked Questions
What can FlowRunner do with PromptLayer?
FlowRunner agents can run List Prompt Templates, Get Prompt Template, and Publish Prompt Template in PromptLayer, plus 17 more actions.
Does connecting PromptLayer to FlowRunner require OAuth?
No. PromptLayer connects to FlowRunner with an API key, no OAuth flow required.
Can PromptLayer trigger a FlowRunner workflow automatically?
PromptLayer doesn't currently expose triggers in FlowRunner. It connects as an action step inside workflows started by another trigger.
Start building with PromptLayer
$100 in credits. No card required. Connect in minutes.