> ## Documentation Index > Fetch the complete documentation index at: https://deepline.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Recover a tool execution > Keep one tool execution across retries, disconnects, and worker restarts using a caller-owned idempotency key. Give each logical tool execution one idempotency key and save it before sending the request. Reuse that key, tool, and input after a timeout. Deepline returns the saved result or follows the original provider job without launching another one. A new key requests new work and can incur another charge. Recovery is opt-in. Existing calls without a key keep their current behavior. The SDK checks that the server supports recovery before submitting keyed work. The normal call still waits for its result. You know the key before opening the connection, so Deepline does not need to return early to give you a recovery ID. If the connection drops, reuse that key. Lookup reads saved state; repeating the original execute request resumes waiting when recovery is possible. Synchronous tools can replay their saved response. Async tools must declare a recoverable launch/status/results lifecycle and run in completion-waiting mode. Keyed launch-only requests, including `wait_for_completion: false`, are rejected before provider dispatch. Tools without that lifecycle return `IDEMPOTENCY_NOT_SUPPORTED`; the error explains the supported alternative. ## TypeScript Use a caller-owned key when a worker may restart. Save both the key and the original input in your application's job record before calling Deepline. ```ts theme={null} import { DeeplineClient } from 'deepline'; const client = new DeeplineClient(); const idempotencyKey = 'sourcing-run-42:company-17'; const input = { domain: 'example.com', currentJobTitle: 'CEO' }; const result = await client.executeTool('datagma_find_people', input, { idempotencyKey, }); console.log(result.billing); console.log(result.executionRecovery); ``` The high-level `Deepline.connect()` context's `tools.execute(toolId, input, options)` accepts the same recovery options. For a short-lived script, `{ recover: true }` generates a key. Save a caller-owned key when recovery must survive the script itself. To persist a generated key before any network request, provide an awaited `onExecution` callback: ```ts theme={null} import { Deepline } from 'deepline'; declare const jobStore: { saveKey(key: string): Promise }; const deepline = await Deepline.connect(); await deepline.tools.execute( 'bounceban_verify_bulk', { emails: ['person@example.com'], }, { recover: true, onExecution: async ({ idempotencyKey }) => { await jobStore.saveKey(idempotencyKey); }, recoveryTimeoutMs: 60_000, }, ); ``` If the callback fails, no execution is sent. Recoverable network failures and active executions are retried with the same key and input. After the first attempt needs recovery, the default reconnect budget is 15 minutes; `recoveryTimeoutMs` changes it. The first request keeps its normal `timeout`. Exhausting the local wait returns `EXECUTION_RECOVERY_TIMEOUT` with the key in `publicDetails`. It does not cancel the provider job. Resume later with the same key. The SDK retries connection failures between your client and Deepline, and `EXECUTION_IN_PROGRESS` responses. A terminal provider error is returned to the caller; recovery does not repeatedly submit a finished failed execution. ### Inspect after a lost response ```ts theme={null} import { Deepline } from 'deepline'; const deepline = await Deepline.connect(); const execution = await deepline.executions.getByKey( 'sourcing-run-42:company-17', ); console.log(execution.executionRecovery.state); console.log(execution.requestId); // Original billing request ID. console.log(execution.responseStatus); if (execution.response) console.log(execution.response); ``` Lookup observes the execution. To resume a recoverable provider job, repeat `tools.execute` with the original tool, input, and key. The SDK's recovery path does this using the original key. It never substitutes a new key after an error. ## CLI Supply a key when your scheduler already owns an execution identity: ```bash theme={null} deepline tools execute bounceban_verify_bulk \ --input @emails.json \ --idempotency-key sourcing-run-42:batch-1 \ --json deepline executions get \ --idempotency-key sourcing-run-42:batch-1 \ --json ``` `emails.json` contains the tool's normal input, for example: ```json theme={null} { "emails": ["person@example.com"] } ``` Use `--recover` to generate a key before dispatch. The CLI reports it on stderr so stdout remains machine-readable JSON. Keep that key to recover the same execution from another process. Running a fresh command with `--recover` reuses its saved key while an identical invocation is unfinished. After the command successfully writes its output, that pending record is removed; a subsequent `--recover` invocation starts new work. Use `--idempotency-key` for an explicit cross-process or cross-machine identity. `--recovery-timeout-ms` changes the recovery wait budget without changing execution identity. If local output generation fails after the server completes, the pending key is retained. Retry with the same input and response format to retrieve the saved result. For the same response format, changing a file destination does not create a new execution. ## Raw HTTP Send `Idempotency-Key` with the normal execute request: ```bash theme={null} curl --fail-with-body "$DEEPLINE_HOST_URL/api/v2/integrations/datagma_find_people/execute" \ -H "Authorization: Bearer $DEEPLINE_API_KEY" \ -H 'Content-Type: application/json' \ -H 'Idempotency-Key: sourcing-run-42:company-17' \ --data '{"payload":{"domain":"example.com","currentJobTitle":"CEO"}}' ``` Look up the same key in the same workspace: ```bash theme={null} curl --fail-with-body "$DEEPLINE_HOST_URL/api/v2/executions/by-key/sourcing-run-42%3Acompany-17" \ -H "Authorization: Bearer $DEEPLINE_API_KEY" ``` The lookup route returns `X-Deepline-Idempotency-Supported: true`, including for an unused key's `404 EXECUTION_NOT_FOUND`. Check this before first dispatch when integrating with a server whose recovery support is unknown. A generic 404 from an older server does not establish support. ## Key and input rules | Rule | Behavior | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | Key length | 1–200 ASCII characters. | | Allowed characters | Letters, digits, `.`, `_`, `:`, and `-`. | | Equality | Case-sensitive. Whitespace is rejected, never trimmed. | | Scope | Keys are unique within a workspace and bound to the original caller authorization identity. A key is not a credential. | | Same key, same request | Observe, resume, or replay the original execution. | | Same key, different request | `409 IDEMPOTENCY_INPUT_MISMATCH`; no new provider call. | | New key, same request | A separate execution that may be charged again. | The server hashes the canonical request with SHA-256. Object property order does not create a conflict; array order and values do. The fingerprint includes the tool and execution-affecting options. Keep the original payload when retrying, including wait behavior and metadata. Resume through the same SDK, CLI, or HTTP call shape. Response format and metadata headers also participate in comparison, so switching between the high-level SDK's dataset response and the raw HTTP response can conflict. Key lookup can inspect the saved response without resubmitting those options. Use the original API key or caller identity for recovery. Another caller in the same workspace cannot read the execution; attempting to submit the same key from that caller conflicts. Rotating an API key does not transfer its saved executions to the new key. The key stays short even for a large batch. Send the normal payload once per HTTP attempt; the server computes its digest. A digest cannot reconstruct your input, and idempotency does not increase HTTP or provider batch limits. `--input @file` reads local JSON; it does not enable an unsupported provider file-upload action. ## States and errors | State or code | Meaning | Caller action | | -------------------------------------- | --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | `running` | The original execution is active or has a recoverable provider job. | Keep the same key; inspect or resume. | | `EXECUTION_IN_PROGRESS` (409) | Execution or delivery is still in progress, including a temporary read failure. | Wait or resume later with the same key. | | `EXECUTION_RECOVERY_UNAVAILABLE` (503) | The recovery record or saved response cannot be read temporarily. | Retry lookup with the same key; this does not authorize new work. | | `EXECUTION_RECOVERY_TIMEOUT` (SDK/CLI) | The caller's recovery wait budget ended. | Inspect or resume later with the same key; the provider job is not cancelled. | | `completed` | The original response was durably saved. | Consume the returned response, including its billing fields. | | `EXECUTION_NOT_FOUND` (404) | This workspace has no execution for the key. | Submit the original request with this key. | | `IDEMPOTENCY_KEY_INVALID` (400) | The key violates the rules above. | Correct it before submitting work. | | `IDEMPOTENCY_INPUT_MISMATCH` (409) | The key is already bound to different input. | Recover the original request; use a new key only for intentional new work. | | `IDEMPOTENCY_KEY_EXPIRED` (410) | The original result's replay window ended. | Retrieve results from your own durable storage. Do not treat expiry as permission to retry paid work. | | `EXECUTION_OUTCOME_UNKNOWN` (409) | Dispatch may have reached the provider, but Deepline cannot safely recover its outcome. | Retain the key and investigate; automatic relaunch is refused. | | `IDEMPOTENCY_NOT_SUPPORTED` (422) | The server or requested execution mode cannot provide this recovery contract. | Follow the error's supported alternative before submitting work. | Completed responses are replayable for 24 hours after completion. The key's record remains after replay expiry, so reusing an expired key produces an explicit error. Active and ambiguous executions never age into permission to launch another call. Save results in your own durable store if you need them beyond the replay window. An idempotency key cannot guarantee an external provider's outcome when the provider accepts work and the process dies before its job ID can be saved. Deepline preserves that uncertainty and refuses duplicate dispatch. Once a recoverable async job ID is durably saved, recovery follows that job. ## Billing and Plays Replay returns the original response and its Deepline billing fields. It does not create a new paid execution. Recovery state describes execution delivery; it does not change billing finality or turn missing billing into a zero charge. Use the billing status and amount in the execution response. For a recovered async job, the completed response refers to the original execution's billing identity. Status and result-fetching calls do not replace the parent execution's cost. Replaying a saved response does not refresh its settlement status: a saved `queued` observation can remain `queued` even after the charge posts. To read current usage for the result above, use its `job_id` with the client's exact billing lookup: ```ts theme={null} import { DeeplineClient } from 'deepline'; const client = new DeeplineClient(); declare const requestId: string; // Use the original response's job_id or execution.requestId. const usage = await client.billing.usageEvent(requestId); console.log(usage.status, usage.credits, usage.billing_outcome_reason); ``` The HTTP equivalent is `GET /api/v2/usage/events?request_id=`. This reads usage; it does not resume execution. A missing event or an unset credit amount does not establish a zero charge. Use an explicit recorded amount, including zero and its outcome reason, when available. After a lost response, `executions.getByKey(key)` also returns `requestId`, the same server-owned ID as the original response's `job_id`. You can pass it to `client.billing.usageEvent()` while the execution is running or its result is unknown. This lets you investigate billing without resubmitting paid work. Idempotency is scoped to one execution. It does not impose a shared spending limit across different keys or concurrent workers. Inside a Play, continue using `ctx.tools.execute({ id, tool, input })`. Plays already own run and call identities. A Play call's `id` is not a public API idempotency key. Use the Play's run ID and run APIs to recover the workflow; this direct-tool API does not change Play authoring syntax.