--- title: "Apify AI Website Crawler Integration" description: "Apify's Website Content Crawler turns an entire site into clean Markdown built for LLM, RAG, and vector-database pipelines. Workflows start crawls with page and spend limits, wait for the run to finish, and pull the results into a knowledge base." url: https://flowrunner.ai/integrations/apify-ai-website-crawler date_modified: 2026-09-04T14:59:52-07:00 --- # Apify AI Website Crawler [Developer Tools](https://flowrunner.ai/integrations/category/developer-infrastructure) Apify's Website Content Crawler turns an entire site into clean Markdown built for LLM, RAG, and vector-database pipelines. Workflows start crawls with page and spend limits, wait for the run to finish, and pull the results into a knowledge base. [Verified](https://flowrunner.ai/integrations/verified "What does verified mean?") · 4 triggers · 17 actions · API key · available [Apify AI Website Crawler website](https://apify.com/) · [Platform Documentation](https://apify.com/apify/website-content-crawler) · Capability data verified 2026-08-25 1. The weekly refresh of the support knowledge base starts 2. Get Crawl Options reads the actor's live input schema so the flow sets bounds the actor actually accepts today 3. Get Monthly Usage reports how much of the account's credit remains before a single page is fetched 4. Start Crawl runs the site with Max Pages and Max Total Charge set, and returns a run id immediately 5. Get Crawl Results reads the finished dataset asking only for the url and markdown fields, and the page count is compared against the last known good crawl 6. The difference in coverage posts to the support channel with the run log attached when pages are missing 7. When the crawl timed out or came back short, a person decides between resurrecting it, starting over, and ingesting what is there ## What This Integration Enables Apify's Website Content Crawler does one thing that a general purpose scraper does not: it returns a site as clean Markdown, stripped of navigation and boilerplate, in the shape a retrieval pipeline actually wants. That makes this connector a corpus builder rather than a data extractor. The distinction matters, because the output of a crawl here does not go into a spreadsheet where a person will read it. It goes into a vector store, and from there into the answers a support assistant gives to customers. The connector treats a crawl as a job rather than a request, which is the honest shape. Start Crawl returns a run id and the On Crawl Succeeded trigger delivers the dataset id when it finishes, so a long crawl does not need a polling loop or a synchronous call that gets cut off partway. Abort Crawl Run stops one that turned out wrong, Resurrect Crawl Run continues a failed or timed out run into the same dataset rather than paying for the whole site again, and Get Crawl Run Log is where a disappointing crawl explains itself: blocked requests, a robots.txt refusal, a selector that never appeared. Those are [automation exceptions](https://flowrunner.ai/concepts/automation-exceptions) with real consequences downstream, which is why this page puts a person on the decision rather than a retry rule. If what you need is a structured record out of a page rather than the page itself, the [Apify LLM Scraper](https://flowrunner.ai/integrations/apify-llm-scraper) connector is the other half of this pair. These two are complements, not alternatives. ### Without FlowRunner **The corpus nobody audits**: A knowledge base gets built once, then quietly ages while the assistant answering from it sounds just as confident **Unbounded crawls**: A crawl pointed at a large site with no page limit keeps going until somebody notices the bill **Failure that does not look like failure**: A run that stopped early still returns pages, so a pipeline treats a partial site as the whole site ### With FlowRunner **Coverage compared, not assumed**: Each refresh is measured against the previous one, and a drop in pages is reported before anything is embedded **Spending bounded up front**: Max Pages and a hard charge ceiling are set on every crawl, and the plan's remaining credit is read before it starts **Partial runs handled deliberately**: A timed out crawl becomes a decision with named options rather than a silent half corpus ## Use Case Scenarios ### A support assistant whose corpus is measured every week Every week the agent starts a crawl of the product documentation with Max Pages set from the last run's count plus a margin, and Max Total Charge as a hard ceiling. On Crawl Succeeded fires with the dataset id, and Get Crawl Results returns just url and markdown so the payload stays the size a retrieval pipeline needs. The chunks are written into [Pinecone](https://flowrunner.ai/integrations/pinecone) alongside the run id that produced them, so any answer can be traced back to the crawl it came from. When the page count drops against the previous week, the agent reads Get Crawl Run Log and posts what it found to the support channel in [Slack](https://flowrunner.ai/integrations/slack) before anything is embedded. ### A competitor's documentation, turned into a briefing A product team wants to know what changed in a competitor's help center. The agent runs Start Crawl with Max Depth low enough to cover the section that matters, saves the output with Save Crawl Results To FlowRunner File so a large crawl never has to sit in a flow's memory, and diffs the Markdown against the previous archive. Only the changed pages go to a model for summarizing, which keeps both the crawl and the reasoning cheap. The briefing lands in [Notion](https://flowrunner.ai/integrations/notion) with links back to the original URLs. ### Onboarding a customer's own site into their workspace A new customer's public site needs to become searchable inside their account. The agent calls Get Crawl Options first so it configures the actor against the schema Apify publishes right now rather than a remembered one, then Start Crawl with a conservative page limit. When On Crawl Timed Out fires instead of On Crawl Succeeded, the agent does not retry blindly. It reads the run's dataset, counts what was captured, and takes the question to a person, because Resurrect Crawl Run is only cheap while the run's storage is still inside the account's retention window. The approved outcome is stored, the customer is told what was indexed, and the records land in [Supabase](https://flowrunner.ai/integrations/supabase). ## Human-in-Loop Highlight The dangerous moment on this connector is not a failure, it is a timeout. On Crawl Timed Out fires and the run still holds a dataset, because the crawler writes pages as it goes. Three doors are now open and they are not equivalent. Resurrect Crawl Run continues into the same dataset and keeps everything already paid for, but only while that storage sits inside the account's retention window, so it is the cheap option with an expiry date on it. Start Crawl begins again and pays for the whole site a second time. Ingesting what is there costs nothing and is the one that quietly ships a half corpus into the assistant customers talk to. No rule the agent could follow gets this right, because the answer depends on how much of the site is missing and how much it matters. So the agent gathers the evidence instead: the pages captured against the previous run's count, the specific reason from Get Crawl Run Log, the remaining credit from Get Monthly Usage, and how long resurrection stays available. Then it asks the knowledge base owner which of the three to take. The crawl is the agent's work. Deciding what the assistant is allowed to have read is not. Agent processes routinely Detects exception requiring judgment Clear match Continues automatically Ambiguous Routes to human via preferred channel Human decides Agent resumes with decision ## Agent Capabilities 17 actions ### Crawling 2 - **Crawl Website** Crawls a website and returns its pages as clean Markdown, waiting for the crawl to finish. Apify holds a synchronous run open for a few minutes only, so this is for a page or a handful rather than a site. - **Start Crawl** Starts the crawl and returns immediately with a run id. The right shape for a real site, and the operation where Max Pages and Max Total Charge do their work, because nothing else bounds a crawl nobody is watching. ### Runs 6 - **Get Crawl Run** Returns a run's status, timing, cost, and dataset id. Every outcome, including a failure, comes back as a healthy response, so the status field is what a flow branches on. - **List Crawl Runs** Returns this account's runs of the crawler. Old runs disappear once the plan's data retention window passes. - **Get Last Crawl Run** Returns the most recent run, optionally only one that ended in a given state. How a flow reaches results whose run id was never stored. - **Abort Crawl Run** Stops a crawl that is still running. The stop spending button for a crawl pointed at the wrong site, though pages already fetched are still billed. - **Resurrect Crawl Run** Restarts a failed, timed out, or aborted run and continues into the same dataset, which is far cheaper than starting a large crawl over. It cannot revive a run whose storage has passed retention. - **Get Crawl Run Log** Returns the crawler's own log as plain text. Blocked requests, robots.txt refusals, and retry exhaustion appear here and in none of the structured fields. ### Results 5 - **Get Crawl Results** Returns the pages a run crawled, from its dataset. Asking for only the fields a pipeline uses is what keeps a hundred page crawl from arriving as full HTML. - **Get Dataset Items** Returns a dataset's items by its own id rather than through a run. Used when a trigger payload or a stored dataset id is all a flow has. - **Save Crawl Results To FlowRunner File** Downloads a run's results in the chosen format into FlowRunner's own file storage. The right shape for a large crawl, and the practical way to get CSV or XLSX out. - **List Crawl Run Files** Lists the keys in a run's key value store, where saved HTML, screenshots, and downloaded files go when those options were turned on. - **Save Crawl Run File To FlowRunner File** Downloads one record from a run's key value store, such as a saved page or a PDF, into FlowRunner's file storage. ### Actor 2 - **Get Crawl Options** Returns the crawler's published input schema with every option, type, default, and enumeration, read live from Apify so it stays correct when the actor gains an option. - **Get Crawler Info** Returns the actor's own record: version, pricing model, default run options, and statistics. The pricing block is where the cost model is named before anything runs. ### Account 2 - **Get Account** Returns the Apify account the token belongs to, with its plan. The cheapest connection check available. - **Get Monthly Usage** Returns this month's usage and cost, broken down by what produced it. The operation to put in front of a scheduled crawl, so a flow notices it is about to exhaust the plan rather than discovering it afterward. ## Triggers 4 triggers ### Event Triggers 4 - **On Crawl Succeeded** Fires when a crawl finishes successfully, carrying the dataset id the results live in. It removes the polling loop a long crawl otherwise needs, and it fires for every run of this crawler on the account, including ones a person started. - **On Crawl Failed** Fires when a crawl fails. A failed run can still hold most of the site, because pages are written as they are fetched, so the dataset is worth reading before the crawl is treated as lost. - **On Crawl Aborted** Fires when a crawl is stopped deliberately, either in the Apify console or through Abort Crawl Run. Worth handling differently from a failure, because a flow that retries here is fighting whoever pressed the button. - **On Crawl Timed Out** Fires when a crawl hits its run timeout. The event that opens the resurrect, restart, or ingest decision this page's human gate exists for. ## Frequently Asked Questions ### What can FlowRunner do with Apify AI Website Crawler? FlowRunner agents can run Crawl Website, Start Crawl, and Get Crawl Run in Apify AI Website Crawler, plus 14 more actions. ### Does connecting Apify AI Website Crawler to FlowRunner require OAuth? No. Apify AI Website Crawler connects to FlowRunner with an API key, no OAuth flow required. ### Can Apify AI Website Crawler trigger a FlowRunner workflow automatically? Yes. Apify AI Website Crawler supports 4 triggers that can start a FlowRunner workflow automatically. **Work at Apify AI Website Crawler?** This integration exposes Apify AI Website Crawler to AI agents on every FlowRunner plan, including through MCP, at no cost to you. [See what FlowRunner offers integration partners](https://flowrunner.ai/integrations/partners), including how to keep this page current. --- Markdown version of https://flowrunner.ai/integrations/apify-ai-website-crawler. Site index: https://flowrunner.ai/llms.txt