Scrapfly

Scrapfly is a managed web scraping and browser automation platform. Workflows scrape with anti-bot protection and JavaScript rendering, batch and schedule scrapes, take screenshots, extract structured data, and crawl whole sites.

Verified 23 actions API key available

Scrapfly website Platform Documentation Capability data verified 2026-08-25

A recurring data need arrives: this site, these pages, this cadence

Scrape URL runs once against a representative page to confirm the content is genuinely there

Agent reads the upstream status inside the response envelope, because a target's 404 arrives as a successful Scrapfly call

Estimate Scrape Cost reports what the agreed configuration costs per run, without performing it

Get Account reports the concurrency ceiling and current usage, which is where a rejection on this API actually comes from

The per-run cost, the cadence, and the webhook name results will arrive on go to the budget owner

The budget owner approves the standing schedule, or trims the configuration before the first tick

What This Integration Enables

Scrapfly is a managed scraping platform with anti-scraping protection, JavaScript rendering, sessions, screenshots, extraction, and a crawler. What makes it distinct in a workflow is that it is unusually honest about money. The vendor tells you which flags are expensive and says not to enable them preemptively. Rendering JavaScript costs roughly five times a plain fetch. A residential proxy pool costs considerably more than a datacenter one. And then it ships Estimate Scrape Cost, a free operation that reports what a configuration would cost without performing it.

That combination is what a workflow platform should be built on. An agent can hold a whole crawl plan in its head, price it, check the concurrency headroom, and bring a number to the person whose budget it is. This is also why the schedule operations deserve more attention than they usually get: a schedule is a recurring commitment with no caller waiting on the other end, which makes it the easiest thing in the platform to forget about. FlowRunner's human-in-the-loop step is where standing commitments get approved rather than accumulated.

Without FlowRunner

Cost discovered afterward A configuration that looked reasonable turns out to have been the expensive combination all month

Standing jobs nobody owns A schedule set up for one project keeps firing after the project ends, and no line item names it

Success that is not success A crawl reports finished having skipped most of the site, and the flow treats the partial set as complete

With FlowRunner

Cost known before the spend A free estimate reports what a configuration costs, so the decision happens before the loop rather than after

Standing commitments approved once, on purpose Cadence, budget, and delivery route are agreed by a person and recorded with the workflow

Completion read properly Failed, skipped, and stop-reason figures are checked alongside the finished flag before results are used

Use Case Scenarios

A market feed on a cadence somebody agreed to

A pricing or listings feed runs on a cron expression through Create Scrape Schedule. The results do not come back to the call that created it, because a scheduled run has no HTTP caller to return to, so the schedule names a webhook registered in Scrapfly's dashboard and pointed at a FlowRunner hook. Each delivery lands in Snowflake with a movement summary to Slack. Two schedule details are worth knowing before the first tick: turning concurrency off means a run still going when the next tick arrives silently skips that tick, and Pause Scrape Schedule keeps the configuration while Delete Scrape Schedule loses it.
A whole site into a knowledge base, with a ceiling on it

Start Crawl handles deduplication, retries, and budget better than a loop over individual scrapes, which is the vendor's own guidance. The page limit is the spend control, and it multiplies: every page is a scrape, so a crawl configuration with JavaScript rendering on is that cost times the limit. When the crawl finishes, Download Crawl Artifact pulls the whole result into workspace storage as one archive rather than paging through contents hundreds of times, and the extracted text feeds the model step through OpenAI. Before anything is used, the agent reads the failed and skipped counts and the stop reason, because a crawl reports finished having skipped most of a site and the finished flag alone will not say so.
Structured fields without a second billed fetch

Extract Data turns a document into structured data using a saved template, an inline template, an LLM prompt, or one of Scrapfly's trained models. It does not fetch the page: you post the content, and the URL is context for the extractor rather than an instruction. So the efficient shape is either pairing it with a scrape whose content is already in hand, or using the scrape's own extraction parameters to do both in one billed call. A template is deterministic and carries no model cost. A prompt costs more and can vary between runs, which matters when the output becomes a record in Google Sheets that somebody reconciles later.

Human-in-Loop Highlight

Create Scrape Schedule is the operation on this connector that spends money forever. An hourly scrape with JavaScript rendering through a residential proxy is a standing bill that arrives every month whether or not anybody still reads the output, and unlike a batch or a crawl there is no caller sitting there to notice it is still running. That is the whole reason Estimate Scrape Cost exists and is free: it is the only way to know the number before committing to it.

So the agent builds the configuration, prices it, and stops: "Proposed schedule: competitor catalog, 40 URLs, hourly, JavaScript rendering on, residential pool. Estimate Scrape Cost puts one run at 47 credits, which is roughly 34,000 a month. Get Account shows 9 of 20 concurrency slots in use at peak. Results would deliver to the flowrunner-catalog webhook. Approve hourly, drop to every six hours, or turn rendering off and re-estimate?" Nothing about that decision is technical. It is a purchase, made once, that keeps being made every hour until somebody pauses it. The agent's job was to make the number visible before the first tick rather than legible on next month's invoice.

Agent processes routinely

Detects exception requiring judgment

Clear match Continues automatically

Ambiguous Routes to human via preferred channel

Human decides

Agent resumes with decision

Agent Capabilities

23 actions

Scraping

Scrape URL Fetches a page through Scrapfly's proxy network, optionally with a real browser, anti-scraping protection, a scripted scenario, a session, and caching. The response is an envelope: the page is in the content field and the target's own HTTP status sits beside it, so a successful Scrapfly call can carry a 404 from the site. Rendering JavaScript is roughly five times the cost, and the vendor's guidance is to enable protection when a response says it is needed rather than preemptively.
Estimate Scrape Cost Reports what a scrape configuration would cost in credits without performing it. Free, and the only way to price a loop before running it. The step that turns a spend decision into something a person can approve.
Batch Scrape Sends up to a hundred scrape configurations in a single request. Every entry needs a correlation id, because results arrive out of order and it is the only way to match a result to its input. If remaining concurrency is below the number of configurations, the whole batch is rejected before anything runs, so there are no partial successes to reconcile.
Save Scrape To FlowRunner File Scrapes a page and stores its content in FlowRunner's own file storage. The right shape for a large page whose text is going into a later step rather than through the workflow's payload, and for anything binary the target serves.

Schedules

List Scrape Schedules Returns the account's scheduled scrapes with their cron expressions and state. Every schedule spends on its own timetable, which makes this the list to review when a bill grows without a project to attach it to.
Get Scrape Schedule Returns one scheduled scrape with its full configuration and retry policy.
Create Scrape Schedule Sets a scrape configuration to run on a cron expression. A standing commitment: it spends credits indefinitely, and without a webhook name the results go nowhere a workflow can see them, because a scheduled run has no HTTP caller to return to.
Update Scrape Schedule Changes a schedule's configuration, cron expression, or retry policy. Only the fields sent are altered.
Execute Scrape Schedule Runs a schedule's scrape immediately, outside its timetable. The way to test a schedule without waiting for the next tick or to force a refresh after a change. It costs exactly what a scheduled run costs, and the result goes to the schedule's webhook rather than back to the caller.
Pause Scrape Schedule Stops a schedule firing without deleting it, which stops the spend while keeping the configuration and the cron expression. The reversible option, and the one an agent should reach for when something looks wrong.
Resume Scrape Schedule Starts a paused schedule firing again from its next tick. Ticks missed while paused are not made up.
Delete Scrape Schedule Removes a scheduled scrape. The configuration and the cron expression go with it, so this is the irreversible half of the pause and delete pair.

Screenshots

Take Screenshot Renders a page and returns the image as binary data rather than JSON, so a workflow expecting an object gets bytes. The capture parameter takes an element selector as well as a full-page flag, which is how one chart or one table is captured rather than the whole page.
Save Screenshot To FlowRunner File Captures a page and stores the image in FlowRunner's own file storage, returning its URL. The form to use unless a workflow is handling raw bytes itself.

Extraction

Extract Data Turns a document into structured data using a saved template, an inline template, an LLM prompt, or one of Scrapfly's trained models. It does not fetch the page: the content is posted and the URL is context for the extractor, so it pairs with a scrape rather than replacing one.

Crawler

Start Crawl Crawls a whole site with budget, depth, and URL-pattern controls, returning an identifier immediately and continuing in the background. The page limit is the spend control, and each page costs a scrape. The vendor recommends this over a loop of individual scrapes, because it handles deduplication, retries, and budget for you.
Get Crawl Status Returns a crawl's progress: pages visited, skipped, and failed, its state, and its credit use. Reading the failed and skipped counts and the stop reason matters more than reading the finished flag, because a crawl reports finished having skipped most of a site.
List Crawl URLs Returns the URLs a crawl has visited with their status. Filtering by status is how the pages that failed or were skipped are found, which is usually where a disappointing crawl explains itself.
Get Crawl Contents Returns the scraped content of a crawl's pages, in raw HTML, text, or Markdown. Markdown is the form to feed a model.
Get Crawl Contents Batch Returns the content of a specific list of crawled URLs in one call. The efficient follow-up once the set has been narrowed, rather than paging a whole crawl to find a handful of pages.
Download Crawl Artifact Downloads a crawl's whole result as a single archive into FlowRunner's own storage. For a large crawl this is the right shape: one file instead of hundreds of content pages. The default artifact format is a web-archive container rather than JSON.
Cancel Crawl Stops a running crawl. The stop-spending operation when a crawl was pointed at the wrong start URL or given too high a limit. Pages already crawled are billed and remain readable.

Account

Get Account Returns the subscription, usage, and concurrency limits. The concurrency figures are the useful part, because a rejection from this API is a concurrency ceiling rather than a per-minute rate limit, and this is where the ceiling and the current usage are reported. Also the cheapest connection check.

Frequently Asked Questions

What can FlowRunner do with Scrapfly?

FlowRunner agents can run Scrape URL, Estimate Scrape Cost, and Batch Scrape in Scrapfly, plus 20 more actions.

Does connecting Scrapfly to FlowRunner require OAuth?

No. Scrapfly connects to FlowRunner with an API key, no OAuth flow required.

Can Scrapfly trigger a FlowRunner workflow automatically?

Scrapfly doesn't currently expose triggers in FlowRunner. It connects as an action step inside workflows started by another trigger.

Start building with Scrapfly

Free plan, no card required. Connect in minutes.

Start Building Free Talk to us