
Data pipeline orchestration coordinates the steps that turn an input into a usable result. It tracks dependencies and run state, decides which failures may retry, and helps an operator see what happened. The transformation code still has to validate data and deliver the output correctly.
Consider a daily orders report: extract a file, validate its records, calculate a summary, then make that summary available. Starting all four steps at midnight would not enforce their order. Retrying the entire job could also repeat a write that already succeeded. The useful question is how the system behaves when a middle step fails.
This guide uses a small local Prefect workflow to make those decisions visible. It complements the Python ETL pipeline guide, which covers a validated import. Here, the focus is coordination across steps, including when downstream work must stop.
A scheduled script can be the right choice for one short, independent job with a clear exit status, protected outputs, and an owner who receives failure alerts. An orchestrator becomes useful when dependencies, partial reruns, overlapping schedules, and operational visibility are making that script difficult to maintain.
| Approach | What it coordinates | What you still design |
|---|---|---|
| Scheduled script | When to start a command | Dependencies, failure reporting, overlap control, and safe reruns |
| Task queue | Work dispatched to consumers | Business completion, duplicate delivery behavior, and cross-step recovery |
| Workflow orchestrator | Runs, dependencies, state, and configured retry behavior | Data contracts, side effects, output handoff, and operating the service |
These categories overlap. A task queue can support workflows, and an orchestrator can launch work through other execution systems. The Celery and Redis guide covers a queue-based deployment; the n8n hosting guide covers operating that automation service. Choose the operating model around the work you need to recover, rather than adding a tool for a simple timer.
For the orders example, validation must succeed before transformation, and transformation must succeed before publication. The graph is a simple chain: extract, validate, transform, publish. A branching pipeline might process independent partitions concurrently, then wait for all required partitions before producing its final report.
“The task returned” is a weak completion rule if its output is incomplete, inaccessible, or based on the wrong input. Define the result a consumer can use: an identified input version, a passed validation contract, a completed output, and an agreed handoff. Track the business batch separately from the orchestrator’s run ID, since one batch may require several attempts.
Prefect flows are decorated Python functions with tracked execution state. This example calls its tasks directly and sequentially. An exception propagates out of the flow, so later calls are not reached. It does not submit parallel tasks or demonstrate a distributed dependency graph.
Validation scope, November 16, 2026: the example was checked on Windows with Python 3.12.14 and Prefect 3.9.3. Fourteen checks passed: thirteen real-engine workflow checks and one isolated file-handoff check with an injected rename failure. The lab uses a temporary local API and database, synthetic orders, and temporary output files. It does not test cloud deployment, schedules, remote workers, concurrent runs, crash recovery, or performance.
Use a fresh Python 3.12 virtual environment and install the directly pinned package below. Transitive dependencies are not pinned by this command. Preserve a tested dependency lock when preparing a deployment.
python -m pip install prefect==3.9.3
Save this as pipeline.py. The teaching contract accepts 1 to 100 orders with unique nonempty IDs and nonnegative integer cent amounts, up to 1,000,000 cents per order. Input is limited to 64 KiB. These are example limits, not measured capacity or a general JSON validation library.
"""Sequential, single-writer teaching flow; synthetic local orders only."""
import hashlib
import json
import os
from pathlib import Path
from tempfile import NamedTemporaryFile
from prefect import flow, task
from prefect.cache_policies import NO_CACHE
from prefect.runtime import task_run
def record(directory, event):
with (Path(directory) / "trace.txt").open("a", encoding="utf-8") as log:
log.write(event + "\n")
@task(retries=0, cache_policy=NO_CACHE, persist_result=False)
def extract(source, directory):
record(directory, "extract")
with open(source, "rb") as stream:
raw = stream.read(65537)
if len(raw) > 65536:
raise ValueError("input exceeds the example's 64 KiB limit")
return json.loads(raw.decode("utf-8")), hashlib.sha256(raw).hexdigest()
@task(retries=0, cache_policy=NO_CACHE, persist_result=False)
def validate(extracted, directory):
record(directory, "validate")
rows, digest = extracted
if not isinstance(rows, list) or not 1 <= len(rows) <= 100:
raise ValueError("expected 1 to 100 orders")
seen = set()
for row in rows:
if not isinstance(row, dict) or set(row) != {"id", "cents"}:
raise ValueError("expected id and cents fields")
key, cents = row["id"], row["cents"]
if not isinstance(key, str) or not key or key in seen:
raise ValueError("order IDs must be nonempty and unique")
if type(cents) is not int or not 0 <= cents <= 1000000:
raise ValueError("cents must be an integer from 0 to 1000000")
seen.add(key)
return rows, digest
@task(retries=1, retry_delay_seconds=0.1,
cache_policy=NO_CACHE, persist_result=False)
def transform(validated, directory, failures=0):
attempt = task_run.run_count
record(directory, f"transform:{attempt}")
if attempt <= failures:
raise ConnectionError("injected transient failure; no remote call")
rows, digest = validated
return {"source_sha256": digest, "transform_version": 1,
"orders": len(rows), "total_cents": sum(r["cents"] for r in rows)}
@task(retries=0, cache_policy=NO_CACHE, persist_result=False)
def publish(summary, directory):
"""Hand off a completed local file. No concurrent-writer guarantee."""
target = Path(directory) / (summary["source_sha256"] + ".json")
payload = (json.dumps(summary, sort_keys=True) + "\n").encode("utf-8")
if target.exists():
if target.read_bytes() != payload:
raise ValueError("existing output differs; inspect before rerunning")
record(directory, "publish:reused")
return {"status": "reused", "path": str(target)}
temporary = None
try:
with NamedTemporaryFile(dir=directory, suffix=".partial", delete=False) as f:
temporary = Path(f.name)
f.write(payload)
os.replace(temporary, target)
finally:
if temporary is not None:
temporary.unlink(missing_ok=True)
record(directory, "publish:created")
return {"status": "created", "path": str(target)}
@flow(name="local-orders-summary", retries=0, persist_result=False)
def orders_pipeline(source, directory, failures=0):
# Direct task calls are sequential; exceptions propagate to the flow.
extracted = extract(source, directory)
checked = validate(extracted, directory)
summary = transform(checked, directory, failures)
return publish(summary, directory)
Only the transformation task has one retry, giving it at most two attempts. Its injected connection failure performs no network request and has no remote side effect. The short retry delay keeps the demonstration quick. Validation errors are deterministic and get no retry.
Save the following launcher as run_local.py beside it. The launcher starts a child process with a fresh Prefect home and profile path, then uses Prefect’s temporary test harness. It requires a loopback API address, disables analytics and API log forwarding for the lab, and removes the temporary files after the run. Expected failure diagnostics may appear on the error stream.
"""Launch a disposable Prefect lab, without reusing an existing profile."""
import json
import os
from pathlib import Path
import subprocess
import sys
import socket
import time
from tempfile import TemporaryDirectory
def demo(root):
from pipeline import orders_pipeline
rows = [{"id": "a", "cents": 1200}, {"id": "b", "cents": 800}]
source = root / "orders.json"
source.write_text(json.dumps(rows), encoding="utf-8")
good = root / "good"
good.mkdir()
first = orders_pipeline(str(source), str(good), failures=1)
again = orders_pipeline(str(source), str(good))
assert first["status"] == "created" and again["status"] == "reused"
assert len(list(good.glob("*.json"))) == 1
print("Transient failure: retried, then created one output")
print("Same input again: reused the matching output")
bad = root / "bad"
bad.mkdir()
source.write_text('[{"id":"a","cents":-1}]', encoding="utf-8")
state = orders_pipeline(str(source), str(bad), return_state=True)
assert state.is_failed() and not list(bad.glob("*.json"))
assert (bad / "trace.txt").read_text().splitlines() == ["extract", "validate"]
print("Invalid input: failed validation; no output published")
def child(mode):
from prefect.testing.utilities import prefect_test_harness
from prefect.settings import PREFECT_API_URL
from urllib.parse import urlparse
with TemporaryDirectory(prefix="orchestration-data-") as temp:
with prefect_test_harness(server_startup_timeout=90):
url = PREFECT_API_URL.value()
assert urlparse(url).hostname == "127.0.0.1", url
if mode == "test":
import test_pipeline
test_pipeline.run(Path(temp), url)
else:
demo(Path(temp))
endpoint = urlparse(url)
# Allow the local listener to finish closing after server termination.
stopped = False
for _ in range(50):
with socket.socket() as probe:
probe.settimeout(0.2)
stopped = probe.connect_ex((endpoint.hostname, endpoint.port)) != 0
if stopped:
break
time.sleep(0.1)
assert stopped, "the lab API did not stop"
if mode == "test":
report_path = Path(__file__).with_name("test-results.json")
report = json.loads(report_path.read_text(encoding="utf-8"))
report["temporaryApiStopped"] = stopped
report_path.write_text(json.dumps(report, indent=2), encoding="utf-8")
if __name__ == "__main__":
mode = "test" if "--test" in sys.argv else "demo"
if "--child" in sys.argv:
child(mode)
else:
with TemporaryDirectory(prefix="orchestration-prefect-") as temp:
env = {k: v for k, v in os.environ.items()
if not k.startswith("PREFECT_")}
env.update(PREFECT_HOME=temp,
PREFECT_PROFILES_PATH=str(Path(temp) / "profiles.toml"),
PREFECT_SERVER_ANALYTICS_ENABLED="false",
PREFECT_SERVER_API_HOST="127.0.0.1",
PREFECT_API_URL="", PREFECT_API_KEY="",
PREFECT_LOGGING_LEVEL="ERROR",
PREFECT_LOGGING_TO_API_ENABLED="false",
PREFECT_SERVER_EPHEMERAL_ENABLED="true")
args = [sys.executable, "-X", "utf8", str(Path(__file__).resolve()),
"--child"] + (["--test"] if mode == "test" else [])
result = subprocess.run(args, env=env, cwd=temp,
capture_output=True, text=True, encoding="utf-8",
creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0))
print(result.stdout, end="")
print(result.stderr, end="", file=sys.stderr)
raise SystemExit(result.returncode)
Run the demonstration from that directory:
python run_local.py
The observed summary was:
Transient failure: retried, then created one output
Same input again: reused the matching output
Invalid input: failed validation; no output published
The first run creates one summary after its simulated transient failure. A second run with the same input bytes reuses the matching output. Invalid data then fails validation in a separate output directory, without calling transformation or publication. The file named trace.txt records step order inside the temporary lab; it is a test aid, not a durable audit service.
A retry means another attempt. It does not mean a previous attempt did nothing. If a remote write succeeds but its response is lost, a retry can repeat that write. Use an idempotency key, a destination constraint, or another protocol appropriate to the actual sink. Verify the result when the outcome is uncertain.
Prefect supports task-specific retry budgets and conditions. Apply them where a failure is plausibly temporary, with delay and jitter appropriate to the service. Do not repeatedly submit bad credentials or malformed input. A flow-level retry also deserves separate review because it can revisit work that already completed.
The example turns off task caching and result persistence. Its repeat-input behavior comes from application code: a source SHA-256 selects the output name, and the existing output must exactly match the newly calculated bytes. A changed source, including changed whitespace, has a different identity. A conflicting existing output causes failure and is preserved for inspection.
This is a sequential, single-writer file protocol. Before handing off a new output, the program writes and closes a temporary file in the same directory, then renames it. That is not a distributed transaction or an exactly-once guarantee. Concurrent writers, process termination, power loss, external modification, and remote object uploads need separate handling and tests. The example does not synchronize output data to durable media.
A workflow needs time limits at the operations that can block: connection establishment, reads, database statements, and remote commands. An orchestrator’s timeout state alone does not prove that an external operation stopped or rolled back. Decide how to cancel work and how to reconcile effects after a timeout. The local example does not exercise timeout behavior.
Set concurrency around the constrained resource. Ten independent tasks may fit the CPU budget but overwhelm a source API or destination database. Also decide whether two runs may process the same business batch. A retry budget, a worker count, and a destination write lock solve different problems.
A backfill processes an earlier data interval. Give it explicit inputs and output ownership rather than substituting today’s date inside yesterday’s job. Record the code version, schema version, input identity, and intended interval. Test a small range before a large replay, and decide whether it may replace existing results or must produce a separately reviewed version.
For operational alerts, watch both failures and missing success. A process that never starts may produce no failure log. Define how late a result can be, who receives the alert, and what information they need to rerun or reconcile it. Protect secrets separately from source code and avoid copying sensitive payloads into logs or orchestration metadata.
A production design has several things to operate: the orchestration service and its database, the processes that do the work, and the destinations that hold completed outputs. Their lifecycles differ. Deleting a disposable worker should not remove the only copy of the run history or the only completed business result.
In Prefect’s worker-based deployment model, workers poll work pools and submit runs to the configured execution environment. The local test harness above is not that deployment. It intentionally discards its state and data when finished. Before self-hosting a persistent service, plan access control, network exposure, upgrades, database backup and restore, and worker recovery.
For object storage, upload a completed file under a deliberate key and provide a separate way for consumers to identify usable results. Verify the uploaded content and any completion signal under failure conditions. A successful local rename does not establish those remote guarantees, and an object store should not be assumed to behave like the local filesystem.
Compute with Hivenet supplies CPU infrastructure for customer-operated workloads. An orchestration service and its workers are software you configure and maintain. This guide does not describe a managed Prefect offering. Object Storage with Hivenet supplies S3-compatible storage for completed pipeline files; the example has not connected to it.
Before choosing capacity, collect a representative batch, peak memory use, runtime, schedule, simultaneous run count, and output retention requirements. Budget separately for the control service and the work it launches. Use those measurements to test a CPU instance, then validate the actual storage handoff and recovery procedure before scheduling unattended production runs.
Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.