Apache Airflow Data Workflows

Pathrule2 Rules • 2 Memories • 2 Skills

Airflow schedules task instances for logical data intervals, not arbitrary scripts at wall-clock time. Jobs become unreliable when they read today's date at runtime, pass large payloads through orchestration metadata, mix side effects across retries, or backfill into the same unpartitioned destination. This pattern constrains interval-driven and idempotent tasks, records DAG parsing and data-passing decisions, and provides separate backfill and incident-triage procedures. It differs from general background queues because Airflow owns dependency graphs, historical intervals, task state, reruns, and data-pipeline orchestration rather than low-latency work distribution.

Suggested path map

Pathrule places each piece on the matching path, so your assistant only sees it where it belongs. This is the scoping you get on import; you can adjust it in your workspace.

/ workspace root

plan-airflow-backfill

triage-airflow-failure

src/

tasks/

Derive every read and write from the logical data interval

Make every task safe to retry and clear

dags/

DAG parsing is pure, fast configuration

Airflow metadata carries references, not datasets

Rules

Derive every read and write from the logical data interval/src/taskshighstrictUse the scheduler-provided interval and run identity for partitions, queries, output keys, and observability.

1	A task may run late, retry tomorrow, or backfill last year. Reading the process clock makes the same task instance operate on different data depending on when infrastructure executes it.
2
3	- Accept interval start, interval end, logical date, and run identity as explicit task inputs rather than calling the current date for partition choice.
4	- Query sources with closed or half-open boundaries that prevent overlap and gaps across adjacent intervals.
5	- Write to a deterministic partition or operation key derived from the logical interval and transformation version.
6	- Include DAG, task, run, map, attempt, and interval identity in logs and data-quality evidence so reruns remain attributable.
7
8	See /dags for the adjacent decision or procedure that completes this constraint.

Make every task safe to retry and clear/src/taskshighstrictSeparate outputs by logical identity, commit atomically, and move external effects behind idempotent keys.

1	Airflow retries and operators can clear task instances manually. A task that appends blindly or sends an unkeyed notification produces duplicates even though orchestration is behaving correctly.
2
3	- Stage output for the task instance and publish it through an atomic replace, partition overwrite, merge, or committed manifest.
4	- Use stable idempotency keys for remote APIs, notifications, and messages and record their outcomes outside transient worker memory.
5	- Do not mark orchestration success until output completeness and required quality checks are durable.
6	- Define cleanup for failed staging data without deleting a prior successful interval or another concurrent run.
7
8	See /tests/dags for the adjacent decision or procedure that completes this constraint.

Memories

DAG parsing is pure, fast configuration/dagsConstruct task graphs without fetching production data, making network calls, or performing expensive dynamic discovery in module import.

1	Schedulers repeatedly parse DAG files. Top-level database queries, cloud listing, secrets calls, or large computation slow the control plane and can make a temporary dependency outage hide the entire workflow.
2
3	- Keep top-level code limited to imports, constants, lightweight configuration, and deterministic graph construction.
4	- Move source discovery and data-dependent branching into tasks or approved dynamic mapping based on bounded task output.
5	- Load secrets at task execution unless graph construction genuinely needs a non-secret configuration value.
6	- Test that importing the DAG module performs no external I/O and remains within the scheduler's parsing budget.
7
8	See /tests/dags for the rule or workflow that puts this decision into practice.

Airflow metadata carries references, not datasets/dagsPass small serializable control values and store real data in an external system with versioned locations.

1	Orchestration metadata is optimized for task coordination, not large tables, files, or model objects. Passing bulk data through it burdens the metadata database and web interface.
2
3	- Return object locations, partition identifiers, row counts, checksums, schema versions, and quality summaries from tasks.
4	- Store datasets in the warehouse, object store, or database appropriate to their format and retention.
5	- Keep control payloads bounded and free of secrets or large personal records because metadata is widely visible to operators.
6	- Make downstream tasks verify the referenced object's identity and completeness rather than trusting a path string alone.
7
8	See /src/tasks for the rule or workflow that puts this decision into practice.

Skills

plan-airflow-backfill/rootBound, stage, observe, pause, and reconcile historical Airflow intervals without overwhelming shared dependencies.

1	---
2	name: plan-airflow-backfill
3	description: Plan and execute an Airflow backfill after logic repair, late data, or a new destination.
4	---
5
6	# Plan Airflow Backfill
7
8	Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result.
9
10	1. Define the exact DAG version, interval range, expected partitions, source retention, destination behavior, and prior successful state.
11	2. Estimate per-interval reads, writes, connections, memory, and external calls, then cap parallelism through pools and batch windows.
12	3. Run a small representative interval set, compare row counts, checksums, quality rules, and downstream visibility, and prove rerun idempotency.
13	4. Expand in bounded batches while watching scheduler, workers, metadata database, source, destination, and downstream freshness.
14	5. Reconcile every intended interval and output, record gaps or exceptions, and remove temporary concurrency or routing changes after completion.
15
16	## Exit criteria
17
18	The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.

triage-airflow-failure/rootClassify Airflow failures across scheduling, dependencies, workers, code, data, retries, and downstream quality.

1	---
2	name: triage-airflow-failure
3	description: Triage a failed, missing, delayed, or repeatedly retried Airflow task or DAG run.
4	---
5
6	# Triage Airflow Failure
7
8	Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result.
9
10	- [ ] Identify DAG, task, run, interval, map index, attempt, worker, queue, and code version before reading a generic stack trace.
11	- [ ] Separate not-scheduled, queued, worker-start, execution, timeout, infrastructure, dependency, data-quality, and publish failures.
12	- [ ] Inspect upstream task state, pools, concurrency, sensors, external dependencies, and metadata health before rerunning the task.
13	- [ ] Decide whether the failure is transient and idempotent, permanently invalid, or requires code or data repair; do not clear state as the first diagnostic action.
14	- [ ] After repair, rerun the smallest safe interval, verify output and downstream state, then document the signal that would detect recurrence earlier.
15
16	## Exit criteria
17
18	The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.

Why this pattern

AI agents often use the current clock instead of the logical interval, perform network discovery while parsing DAG files, or retry a task that appends duplicate output.

Built for Data teams operating scheduled batch pipelines, dependencies, backfills, and dataset-driven workflows.

Keeps your assistant from:

Writing the wrong partition during a delayed or historical run
Slowing every scheduler parse with network or database calls
Duplicating output after task retry or manual clear
Overloading a shared warehouse through unconstrained parallel backfill

View as markdown

License: Apache-2.0
Version: 1.0.0
Updated: 2026-08-25