API ReferenceREST API Reference

REST API Reference

Trigger pipeline DAGs sequentially, register sophisticated new data contracts, and query monitoring metrics programmatically using standard HTTP JSON workflows.

Asynchronous Trigger Patterns

Sometimes massive Data Pipelines should not be scheduled sequentially by a rigid Cron clock, but triggered flexibly by an external downstream event.

For instance, when a third-party analytics vendor finishes securely uploading a batched CSV to your external SFTP server, their connected system can aggressively fire a webhook to the DataFlow REST API to instantly trigger the ingestion DAG without waiting for the nightly processing window.

Webhook Ingestion Trigger

You must authenticate securely using a heavily restricted ephemeral Bearer token provisioned via the Service Accounts UI. Never embed root API keys into webhook configurations.

POST https://api.dataflow.ai/v1/workspaces/ws_abc123/pipelines/pl_xyz789/trigger

Inspecting Execution States

The POST trigger endpoint strictly returns an immediate HTTP 202 Accepted header, dispensing a highly unique run_id token payload rather than keeping the remote connection open.

Real-Time Polling Architecture

You can subsequently query the /v1/runs/{run_id} endpoint dynamically in a customized polling loop to inspect the resultant PySpark cluster deployment status and row insertions in real-time.