> ## Documentation Index
> Fetch the complete documentation index at: https://deepline.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Firecrawl Crawl: Inputs, Cost & CLI Example
> Crawling a website by following links from a starting URL. Includes inputs, outputs, pricing notes, Deepline CLI examples, and GTM automation guidance.
## Run in Enrichment Spreadsheet
Use this function as a column step in `deepline enrich`.
```bash theme={null}
deepline enrich --input leads.csv --output leads.enriched.csv --with 'result=firecrawl_crawl:{"url":"{{url}}"}' --json
```
Map payload values to spreadsheet columns with `{{column_name}}` placeholders.
## Input Schema
| Name | Type | Required | Default | Description |
| ------------------------------- | ------------------------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------ |
| `payload.url` | `string` | Yes | | Starting URL to crawl |
| `payload.maxDiscoveryDepth` | `integer` | No | | Max link depth to discover |
| `payload.limit` | `integer` | No | | Max pages to crawl (default 10000) |
| `payload.includePaths` | `array` | No | | Only crawl URLs matching these glob patterns |
| `payload.excludePaths` | `array` | No | | Skip URLs matching these glob patterns |
| `payload.scrapeOptions` | `object` | No | | Scrape options for each page |
| `payload.webhook` | `object` | No | | Webhook for completion notification |
| `payload.prompt` | `string` | No | | Natural language crawl guidance |
| `payload.sitemap` | `"skip" \| "include" \| "only"` | No | | Sitemap handling (default "include") |
| `payload.ignoreQueryParameters` | `boolean` | No | | Ignore query parameters in URLs (default false) |
| `payload.regexOnFullURL` | `boolean` | No | | Apply regex patterns to full URL (default false) |
| `payload.crawlEntireDomain` | `boolean` | No | | Crawl the entire domain (default false) |
| `payload.allowExternalLinks` | `boolean` | No | | Follow external links (default false) |
| `payload.allowSubdomains` | `boolean` | No | | Follow subdomain links (default false) |
| `payload.delay` | `number` | No | | Seconds to wait between scrapes |
| `payload.maxConcurrency` | `integer` | No | | Max concurrent scrapes |
| `payload.zeroDataRetention` | `boolean` | No | | Do not store any data (default false) |
| `payload.timeoutMs` | `integer` | No | | Poll timeout in ms (default 30000; larger values are clamped to the Deepline async sync-wait ceiling). |
### Allowed values
| Field | Allowed values |
| ----------------- | ------------------------- |
| `payload.sitemap` | `skip`, `include`, `only` |
Show raw input schema
### Input JSON Schema
```json theme={null}
{
"type": "object",
"description": "Crawl a website starting from a URL (async with polling).",
"properties": {
"url": {
"type": "string",
"description": "Starting URL to crawl"
},
"maxDiscoveryDepth": {
"type": "integer",
"description": "Max link depth to discover"
},
"limit": {
"type": "integer",
"description": "Max pages to crawl (default 10000)"
},
"includePaths": {
"type": "array",
"description": "Only crawl URLs matching these glob patterns",
"items": {
"type": "string"
}
},
"excludePaths": {
"type": "array",
"description": "Skip URLs matching these glob patterns",
"items": {
"type": "string"
}
},
"scrapeOptions": {
"type": "object",
"description": "Scrape options for each page",
"properties": {
"formats": {
"type": "array",
"description": "Output formats (default [{\"type\":\"markdown\"}])",
"items": {
"anyOf": [
{
"type": "string",
"enum": [
"markdown",
"summary",
"html",
"rawHtml",
"links",
"images",
"screenshot",
"json",
"changeTracking",
"branding"
]
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "markdown"
}
},
"required": [
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "summary"
}
},
"required": [
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "html"
}
},
"required": [
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "rawHtml"
}
},
"required": [
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "links"
}
},
"required": [
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "images"
}
},
"required": [
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "screenshot"
},
"fullPage": {
"type": "boolean",
"description": "Capture full page screenshot"
},
"quality": {
"type": "integer",
"description": "Screenshot quality 0-100"
},
"viewport": {
"type": "object",
"description": "Custom viewport size",
"properties": {
"width": {
"type": "integer"
},
"height": {
"type": "integer"
}
},
"additionalProperties": false
}
},
"required": [
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "json"
},
"schema": {
"type": "object",
"description": "JSON schema for structured output",
"additionalProperties": {}
},
"prompt": {
"type": "string",
"description": "Natural language extraction instructions"
}
},
"required": [
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "changeTracking"
},
"modes": {
"type": "array",
"description": "Change tracking modes",
"items": {
"type": "string",
"enum": [
"git-diff",
"json"
]
}
},
"schema": {
"type": "object",
"description": "Schema for tracked changes",
"additionalProperties": {}
},
"prompt": {
"type": "string",
"description": "Instructions for change tracking"
},
"tag": {
"type": "string",
"description": "Tag for this change tracking snapshot"
}
},
"required": [
"type"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"const": "branding"
}
},
"required": [
"type"
],
"additionalProperties": false
}
]
}
},
"onlyMainContent": {
"type": "boolean",
"description": "Only return main content, filtering navs/footers"
},
"includeTags": {
"type": "array",
"description": "Only include these HTML tags",
"items": {
"type": "string"
}
},
"excludeTags": {
"type": "array",
"description": "Exclude these HTML tags",
"items": {
"type": "string"
}
},
"headers": {
"type": "object",
"description": "Custom HTTP headers",
"additionalProperties": {
"type": "string"
}
},
"waitFor": {
"type": "number",
"description": "Wait ms after page load before scraping"
},
"mobile": {
"type": "boolean",
"description": "Use mobile user agent"
},
"timeout": {
"type": "number",
"description": "Request timeout in ms (default 30000, max 300000)"
},
"maxAge": {
"type": "integer",
"description": "Max cache age in ms (default 172800000)"
},
"storeInCache": {
"type": "boolean",
"description": "Store result in cache (default true)"
},
"parsers": {
"type": "array",
"description": "Document parsers (e.g. PDF)",
"items": {
"type": "object",
"properties": {
"type": {
"type": "string",
"description": "Parser type",
"const": "pdf"
},
"mode": {
"type": "string",
"description": "PDF parsing mode",
"enum": [
"fast",
"auto",
"ocr"
]
},
"maxPages": {
"type": "integer",
"description": "Max pages to parse (1-10000)"
}
},
"required": [
"type"
],
"additionalProperties": false
}
},
"actions": {
"type": "array",
"description": "Browser actions to execute before scraping",
"items": {
"type": "object",
"properties": {
"type": {
"type": "string",
"description": "Browser action type",
"enum": [
"wait",
"click",
"write",
"press",
"screenshot",
"scroll",
"pdf",
"scrape",
"executeJavascript"
]
},
"selector": {
"type": "string",
"description": "CSS selector for the element"
},
"milliseconds": {
"type": "number",
"description": "Wait time in milliseconds"
},
"text": {
"type": "string",
"description": "Text to write"
},
"key": {
"type": "string",
"description": "Key to press"
},
"direction": {
"type": "string",
"description": "Scroll direction",
"enum": [
"up",
"down"
]
},
"amount": {
"type": "number",
"description": "Scroll amount in pixels"
},
"script": {
"type": "string",
"description": "JavaScript to execute"
},
"fullPage": {
"type": "boolean",
"description": "Full-page screenshot"
},
"format": {
"type": "string",
"description": "PDF page format",
"enum": [
"A0",
"A1",
"A2",
"A3",
"A4",
"A5",
"A6",
"Letter",
"Legal",
"Tabloid",
"Ledger"
]
},
"landscape": {
"type": "boolean",
"description": "PDF landscape orientation"
},
"scale": {
"type": "number",
"description": "PDF scale factor (0.1-2)"
}
},
"required": [
"type"
],
"additionalProperties": false
}
},
"location": {
"type": "object",
"description": "Geographic location settings",
"properties": {
"country": {
"type": "string",
"description": "Country code (e.g. \"US\")"
},
"languages": {
"type": "array",
"description": "Language codes",
"items": {
"type": "string"
}
}
},
"additionalProperties": false
},
"proxy": {
"type": "string",
"description": "Proxy mode",
"enum": [
"basic",
"enhanced",
"auto"
]
},
"blockAds": {
"type": "boolean",
"description": "Block ads during scraping"
},
"removeBase64Images": {
"type": "boolean",
"description": "Strip base64 images from output"
},
"skipTlsVerification": {
"type": "boolean",
"description": "Skip TLS certificate verification"
}
},
"additionalProperties": false
},
"webhook": {
"type": "object",
"description": "Webhook for completion notification",
"properties": {
"url": {
"type": "string",
"description": "Webhook URL for notifications"
},
"headers": {
"type": "object",
"description": "Custom headers for webhook requests",
"additionalProperties": {
"type": "string"
}
},
"metadata": {
"type": "object",
"description": "Metadata to include in webhook payloads",
"additionalProperties": {}
},
"events": {
"type": "array",
"description": "Events to trigger webhook (default all)",
"items": {
"type": "string",
"enum": [
"completed",
"page",
"failed",
"started"
]
}
}
},
"required": [
"url"
],
"additionalProperties": false
},
"prompt": {
"type": "string",
"description": "Natural language crawl guidance"
},
"sitemap": {
"type": "string",
"description": "Sitemap handling (default \"include\")",
"enum": [
"skip",
"include",
"only"
]
},
"ignoreQueryParameters": {
"type": "boolean",
"description": "Ignore query parameters in URLs (default false)"
},
"regexOnFullURL": {
"type": "boolean",
"description": "Apply regex patterns to full URL (default false)"
},
"crawlEntireDomain": {
"type": "boolean",
"description": "Crawl the entire domain (default false)"
},
"allowExternalLinks": {
"type": "boolean",
"description": "Follow external links (default false)"
},
"allowSubdomains": {
"type": "boolean",
"description": "Follow subdomain links (default false)"
},
"delay": {
"type": "number",
"description": "Seconds to wait between scrapes"
},
"maxConcurrency": {
"type": "integer",
"description": "Max concurrent scrapes"
},
"zeroDataRetention": {
"type": "boolean",
"description": "Do not store any data (default false)"
},
"timeoutMs": {
"type": "integer",
"description": "Poll timeout in ms (default 30000; larger values are clamped to the Deepline async sync-wait ceiling)."
}
},
"required": [
"url"
],
"additionalProperties": false
}
```
## Output Schema
| Name | Type | Required | Default | Description |
| ------------- | -------- | -------- | ------- | ---------------------------------------------- |
| `result.data` | `object` | Yes | | Provider response payload. |
| `result.meta` | `record` | No | | Additional response metadata (status, paging). |
Show raw output schema
### Output JSON Schema
```json theme={null}
{
"type": "object",
"description": "Crawl job result.",
"properties": {
"data": {
"type": "object",
"description": "Provider response payload.",
"properties": {
"status": {
"type": "string"
},
"total": {
"type": "number"
},
"completed": {
"type": "number"
},
"expiresAt": {
"type": "string"
},
"next": {
"type": "string"
},
"data": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": {}
}
}
},
"required": [
"status"
],
"additionalProperties": false
},
"meta": {
"type": "object",
"description": "Additional response metadata (status, paging).",
"additionalProperties": {}
}
},
"required": [
"data"
],
"additionalProperties": false
}
```
## Advanced: Direct CLI
Use direct execution for single payload debugging.
```bash theme={null}
deepline tools execute firecrawl_crawl --payload '{
"url": "string"
}' --json
```
### CLI flags
| Flag | Description |
| --------------------------- | --------------------------------------------------- |
| `--json` | Print machine-readable output. |
| `--wait` | Wait for terminal provider status when supported. |
| `--debug` | Enable wait mode with additional status/log output. |
| `--wait-timeout SECONDS` | Max seconds to wait in wait mode. |
| `--poll-interval SECONDS` | Polling interval in seconds during wait mode. |
| `--timeout SECONDS` | Request timeout in seconds. |
| `--connect-timeout SECONDS` | Connection timeout in seconds. |
## Provider API Context
Show provider reference (optional)
```md theme={null}
```
## Cost
* Pricing model: `per_page` (per page).
* Estimated Deepline credits: `0.02` per pricing unit.
* Billing mode: `post_deduct`.