Apache Airflow for B2B Data Pipelines

When DAGs beat cron for B2B sync

Use a DAG orchestrator when jobs have dependencies, fan-out/fan-in, different SLAs per step, or must skip downstream work when upstream data is empty or late. Classic B2B cases: ERP → staging → validation → OLTP upsert → warehouse load; multi-source customer master merge; month-end extracts that must not overlap; backfills that replay a date range safely. Cron remains fine for a single idempotent task with no dependents: send digest email, refresh a materialised cache, ping a health check.

• DAG win: multi-step ERP/warehouse pipelines with clear ownership
• DAG win: retries and alerting per task, not only per crontab line
• Cron win: one job, one schedule, trivial failure mode
• Neither: request-path work that belongs in an application queue

What Airflow is good at (in practice)

Airflow schedules and monitors batch/workflow DAGs. It is not your application runtime for user clicks, and it is not a streaming engine. Treat it as the control plane for data and integration workflows. Strengths: explicit dependencies, retries with backoff, SLAs/alerts, backfill, and a UI operators can open during an incident. Managed offerings (MWAA, Composer, Astro, etc.) reduce ops burden versus self-hosting. Weaknesses: operational weight for small teams, Python DAG sprawl, and a temptation to put business logic inside operators instead of versioned application services.

ERP and warehouse pipeline patterns

Separate extract, validate, load-to-app, and load-to-warehouse. Validation failures should fail the DAG loudly with artifacts (row counts, sample rejects), not silently load partial truth into customer-facing tables. Idempotent loads keyed by natural ERP IDs protect retries. Align with ERP integration design and cutover discipline when historical backfill is in scope. Warehouse steps often speak different dialects (Teradata, BigQuery, Snowflake, Redshift). Keep SQL in reviewed files; see working with enterprise warehouses for ownership boundaries between product and data teams.

Retries, idempotency, and poison data

Retries without idempotency create duplicate orders, duplicate invoices, or duplicate tickets. Every task that writes must define its upsert key and partial-failure behavior. Quarantine poison rows: park bad records, alert owners, continue or fail based on business rules. Blind 'retry 5 times' on bad master data only delays the incident. Use correlation IDs from DAG run through application logs so support can answer 'did last night's sync finish for tenant X?'

• Document upsert keys per entity
• Dead-letter or quarantine tables for rejects
• Cap concurrent runs that touch the same tenant
• Make backfills a first-class DAG, not a one-off SSH script

Ownership, on-call, and change control

Airflow fails socially when nobody owns DAG breakage after a schema change. Assign product-engineering ownership for app-facing loads and data-platform ownership for warehouse transforms when both exist. Deploy DAGs like code: PR review, CI parse/test, staged rollout. A broken import on Monday morning is a customer incident, not a 'data team ticket'. Tie alerts to observability standards: success/failure, duration, rows processed, lag vs source system. Dashboard the business SLA (data fresh by 07:00 plant local), not only green task squares.

When NOT to use Airflow

Skip Airflow for request/response APIs, per-user interactive jobs, or a handful of simple cron tasks a small team can reason about in the app's worker. Skip it when your 'pipeline' is really an event-driven flow better served by a queue and consumers (order placed → enrich → notify). Airflow can trigger on sensors, but streaming/event platforms fit continuous flows better. Skip self-hosted Airflow if you lack capacity for metadata DB, executors, and upgrades. The orchestrator must not become more fragile than the jobs it runs. For early MVP sync, a well-tested worker + schedule in your app stack may be enough—see MVP prioritization and stack selection.

Alternatives worth comparing

Application queues and schedulers (Sidekiq, BullMQ, Celery beat, cloud scheduler + workers): best when logic already lives in the product and steps are few. Cloud-native orchestrators (Step Functions, Cloud Workflows, Azure Logic/Durable Functions): good when you want managed state machines and tighter cloud IAM integration. ELT-focused tools (dbt + scheduler, managed ingestion) shine for analytics transforms after raw landings; they do not replace ERP writebacks into OLTP. Pick based on who on-calls and where business rules must live—not on conference popularity.

Security, secrets, and tenancy

Pipeline workers often hold ERP and DWH credentials. Use secret managers, short-lived credentials where possible, and least privilege per connection—not one superuser for all DAGs. Multi-tenant SaaS sync must isolate tenant runs or enforce tenant_id in every write. A 'global' job that forgets the filter is a data-leak incident; align with multi-tenant architecture and audit logging. Log what ran, for which tenant, with which code version. Auditors and enterprise buyers will ask.

Delivery risk and contractor handoff

Contractors can ship impressive DAGs that only they understand. Acceptance should include runbooks, idempotency tests, failure injection, and documented ownership—see hiring contractors for B2B. Budget ops time in cost planning: Airflow is not free after the first green demo. Include staging ERP sandboxes in technical discovery.

Next steps

Draw your current sync as boxes and arrows. If you have more than two dependent steps or regular partial failures, evaluate a DAG orchestrator. If you have one box, harden cron/workers first. Related: Postgres vs MongoDB, production readiness, other resources, case studies, book a call, or contact.

FAQ