You're Collecting
Data — But Is It
Actually Arriving?

The problem with cron jobs — you find out at 2am

Most teams start the same way. You need a pipeline: pull data with SQL, clean it with Python, load it into a final table. The solution looks obvious — set up a cron job, schedule the scripts, move on.

In production that approach breaks quietly. A script fails and nobody knows. A dependency breaks and the data is left half-processed. The machine restarts and the cron job never comes back. You walk in the next morning to an empty dashboard.

Cron is a scheduler, not an orchestrator. It runs one script at one time, and that is all. It does not know whether the previous script finished or failed. It will not retry. It will not alert anyone. It does not track dependencies.

What a data pipeline is

A data pipeline is an automated workflow that picks data up from a source, processes it, and loads it into a defined destination. Each step does one specific job, and the steps run in a defined order.

DAG — what a directed acyclic graph is

When a pipeline has many steps, some of which can run in parallel while others must run in sequence, you use a DAG to define which task runs when.

Directed — every task has a direction. Task A leads to Task B; it never runs backward.
Acyclic — no cycles. Task A → Task B → Task A is not allowed, so there is no infinite loop.
Graph — tasks are the nodes, dependencies are the edges.

                    ┌─────────────────────┐
                    │   START (Trigger)   │
                    │   Daily @ 2:00 AM   │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │  extract_raw_sales  │  ← Task A: pull raw CSV from S3
                    └──────────┬──────────┘
                               │
               ┌───────────────┼───────────────┐
               ▼               ▼               ▼
      ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
      │ clean_nulls  │ │ remove_dupes │ │ fix_formats  │  ← Parallel tasks
      └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
             └───────────────┬┘               ─┘
                             │
                             ▼
                  ┌─────────────────────┐
                  │  transform_revenue  │  ← Aggregate by city and by product
                  └──────────┬──────────┘
                             │
                             ▼
                  ┌─────────────────────┐
                  │  validate_output    │  ← Row count check, schema validation
                  └──────────┬──────────┘
                             │
                             ▼
                  ┌─────────────────────┐
                  │  load_to_warehouse  │  ← Load into BigQuery / Redshift
                  └──────────┬──────────┘
                             │
                             ▼
                  ┌─────────────────────┐
                  │  notify_team_slack  │  ← "Daily report ready" message
                  └─────────────────────┘
            

In this DAG the three cleaning steps run in parallel, all at the same time. But transform_revenue does not start until all three cleaning tasks have finished. The orchestrator handles that dependency tracking for you.

If validate_output fails, load_to_warehouse never runs. Corrupt data never reaches the warehouse, and the team gets a Slack alert saying validation failed.

Apache Airflow — Industry Standard Orchestrator

Airbnb built Apache Airflow in 2014 for its own internal data pipelines and open-sourced it in 2016. Today it is the most widely used pipeline orchestration tool in data engineering — Uber, Twitter, Lyft and Flipkart all run on it.

In Airflow you write the DAG as Python code. It ships with a web UI where you can monitor your pipelines, trigger them manually, or retry individual failed tasks.

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

# Default settings — har task pe apply honge
default_args = {
    'owner': 'data-team',
    'retries': 3,                          # Failure pe 3 baar retry karega
    'retry_delay': timedelta(minutes=5),   # 5 minute wait karke retry
    'email_on_failure': True,              # Fail hone pe email
    'email': ['data-team@company.com']
}

# Define the DAG
with DAG(
    dag_id='daily_sales_pipeline',
    default_args=default_args,
    schedule_interval='0 2 * * *',         # Har raat 2 baje
    start_date=datetime(2025, 1, 1),
    catchup=False                          # Ignore past missed runs
) as dag:

    # Task 1 — Extract
    extract_task = PythonOperator(
        task_id='extract_raw_sales',
        python_callable=extract_from_s3    # Tumhara Python function
    )

    # Task 2 — Transform
    transform_task = PythonOperator(
        task_id='transform_revenue',
        python_callable=aggregate_by_city
    )

    # Task 3 — Load
    load_task = PythonOperator(
        task_id='load_to_warehouse',
        python_callable=load_to_bigquery
    )

    # Define the dependency - this is what makes it a DAG
    extract_task >> transform_task >> load_task
    # Extract complete → Transform start → Load start
            

Airflow is powerful, but the setup is heavy. You need a dedicated server, a Postgres or MySQL database, and a scheduler process running separately. For large teams that cost is worth paying. For small projects it is usually over-engineering.

Prefect — Modern Python-Native Alternative

Prefect is a newer orchestration tool built to address Airflow's complexity. Its core pitch is simple: "It's just Python." There is no separate DAG syntax to learn. You write the Python you already know and add decorators.

from prefect import flow, task
from prefect.tasks import task_input_hash
from datetime import timedelta

# @task decorator — is function ko pipeline task banao
@task(retries=3, retry_delay_seconds=60, cache_key_fn=task_input_hash)
def extract_raw_sales():
    # Pull data from S3 or a database
    return fetch_data_from_s3("s3://bucket/sales/")

@task
def transform_revenue(raw_data):
    # Aggregate revenue by city
    return raw_data.groupby("city")["revenue"].sum()

@task
def load_to_warehouse(transformed_data):
    # Load into BigQuery
    write_to_bigquery(transformed_data, table="sales.daily_revenue")

# @flow decorator - this is your pipeline entry point
@flow(name="Daily Sales Pipeline")
def daily_sales_pipeline():
    raw = extract_raw_sales()
    aggregated = transform_revenue(raw)
    load_to_warehouse(aggregated)

if __name__ == "__main__":
    daily_sales_pipeline()        # Locally test kar sakte ho directly
            

Prefect needs no separate configuration file. You write a Python function, add a decorator, and the pipeline is ready. The same code runs locally and in production, and testing is far simpler because these are plain Python functions.

Manual cron scripts vs a data pipeline — the difference in production

Both approaches work in a simple scenario. The difference shows up once the pipeline gets complex, the data volume grows, or something unexpected fails at 2am.

  CRON JOB APPROACH:
  ─────────────────────────────────────────────────────
  step_1.py  ✓   step_2.py  ✓   step_3.py  ✗   [SILENT]
                                     │
                                     └── Script exits with error code 1
                                         No one knows.
                                         step_4, step_5, step_6 — did they run?
                                         Was data partially loaded?
                                         Dashboard empty in the morning.
                                         Manual investigation starts at 9 AM.

  AIRFLOW / PREFECT PIPELINE:
  ─────────────────────────────────────────────────────
  Task 1  ✓   Task 2  ✓   Task 3  ✗
                               │
                               ├── Immediate Slack / Email alert: "Task 3 failed"
                               ├── Retry attempt 1 (after 5 min)  ✗
                               ├── Retry attempt 2 (after 5 min)  ✗
                               ├── Retry attempt 3 (after 5 min)  ✓  ← Fixed itself
                               │
                               ├── Task 4, 5, 6 resume automatically
                               └── Full logs available in UI — exact error line