Why Python is the default language for data engineering
Step into data engineering and one thing becomes obvious fast: every tool, every framework, every pipeline library is written in Python. That is not a coincidence.
The entire data engineering ecosystem is built on top of Python. Once an ecosystem settles on one language, that language becomes a gravity well, and every other option gets pushed to the margins.
SQL-style and programmatic transformations on DataFrames. Cleaning, merging, reshaping — the fastest iteration loop on a single machine.
The Python API for Apache Spark. Parallelize petabyte-scale data across a cluster. Swiggy, Flipkart and Jio all run on PySpark.
Write DAGs in Python and schedule pipelines. Dependency management, retry logic and monitoring are all built in.
Python orchestrates your SQL models. Version control, testing and documentation — the backbone of the modern data stack.
Spotify-born pipeline framework. Long-running batch jobs, file-based dependencies, simple but powerful for data workflows.
Put data quality gates in the pipeline. Define expectations and the pipeline validates them automatically before loading.
What SQL cannot do
SQL is a powerful language, but for one specific job: querying data that is already structured in a table. A data engineer's job is much broader than that.
In the real world, pipelines at a company like Zomato do work where SQL alone falls over completely:
Why not Java, Scala or Go?
It is a fair question, and the answer is not technical. It is about the ecosystem.
Java and Scala are technically the native languages of Apache Spark — Spark itself is written in Scala. Go is extremely fast. So why don't these languages dominate data engineering?
Python ████████████████████████ Pandas, PySpark, Airflow, dbt,
Luigi, Great Expectations,
Prefect, Dagster, Delta Lake,
Kafka-Python, SQLAlchemy...
Java / Scala ██████░░░░░░░░░░░░░░░░░░ Spark (native), Flink
(most tools have no Java SDK)
Go ███░░░░░░░░░░░░░░░░░░░░░ Almost nothing DE-specific
R █████░░░░░░░░░░░░░░░░░░░ Stats/analysis, not pipelines
Doing data engineering in Java or Scala means building most of it yourself. In Python it already exists, it is tested, it is battle-hardened in production, and it comes with community support.
Pandas — what a single machine can do, and where it stops
Pandas is the first tool most people learn in data engineering, for one reason: it is genuinely excellent at what it does.
Pandas is a DataFrame library that loads the entire dataset into one machine's RAM. You then run operations on that data — filter, group, merge, transform. It all processes sequentially on a single core.
CSV / Database / API
│
▼
┌─────────────────────────────────────────┐
│ Your Laptop / Server │
│ │
│ ┌───────────────────────────────────┐ │
│ │ RAM (e.g. 16 GB) │ │
│ │ │ │
│ │ df = pd.read_csv("data.csv") │ │
│ │ ┌─────────────────────────────┐ │ │
│ │ │ Entire Dataset Loaded Here │ │ │
│ │ │ (all rows, all columns) │ │ │
│ │ └─────────────────────────────┘ │ │
│ └───────────────────────────────────┘ │
│ │
│ Single CPU Core — Sequential execution │
└─────────────────────────────────────────┘
│
▼
df.groupby("city").agg({"revenue": "sum"})
→ Processed one row at a time, on one core
import pandas as pd
# Works perfectly for small data
df = pd.read_csv("orders_10k_rows.csv") # 10k rows → loads instantly
df_clean = df[df["status"] == "delivered"]
df_grouped = df_clean.groupby("city").agg({"revenue": "sum"})
print(df_grouped.head())
# Fast, readable, done in 3 lines ✓
# -----------------------------------------
# The same code on a 50 GB file:
df = pd.read_csv("orders_500M_rows.csv") # 500M rows
# MemoryError: Unable to allocate 50.0 GiB for an array
# Your machine has 16 GB RAM.
# Pandas tried to load 50 GB into it.
# It crashed. Pipeline down. Data not processed.PySpark — what distributed processing actually buys you
PySpark is the Python interface to Apache Spark. Spark's core design idea is simple: if the data does not fit on one machine, split it across many machines and have them all work at once.
50 GB Dataset (S3 / HDFS / ADLS)
│
▼
┌─────────────────────────────────────────────────┐
│ Driver Node │
│ (Your PySpark program runs here) │
│ Builds Execution Plan (DAG) │
└────────────────────┬────────────────────────────┘
│ distributes partitions
┌────────────┼────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Worker 1 │ │ Worker 2 │ │ Worker 3 │
│ │ │ │ │ │
│ Part 1 │ │ Part 2 │ │ Part 3 │
│ (17 GB) │ │ (17 GB) │ │ (16 GB) │
│ │ │ │ │ │
│ Process │ │ Process │ │ Process │
│ in │ │ in │ │ in │
│ parallel │ │ parallel │ │ parallel │
└──────────┘ └──────────┘ └──────────┘
│ │ │
└────────────┼────────────┘
▼
Aggregated Result
(collected to driver)
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
# Start a Spark session (connects to cluster)
spark = SparkSession.builder \
.appName("OrderAnalysis") \
.getOrCreate()
# 50 GB file — PySpark doesn't load it into RAM
# It creates a distributed plan across the cluster
df = spark.read.csv("s3://data-lake/orders_500M_rows.csv",
header=True, inferSchema=True)
# This line does NOT execute yet — Lazy Evaluation
df_clean = df.filter(F.col("status") == "delivered")
df_grouped = df_clean.groupBy("city").agg(
F.sum("revenue").alias("total_revenue")
)
# Execution happens only when you call an action
# Spark optimizes the entire plan before running
df_grouped.write.parquet("s3://output/city_revenue/")
# Ran across 10 worker nodes in parallel
# 50 GB processed without any memory errorPySpark has one critical feature that Pandas does not: lazy evaluation.
Pandas (Eager Evaluation):
─────────────────────────
df.filter(...) ← executes immediately, scans all rows
df.groupBy(...) ← executes immediately on full filtered data
df.select(...) ← executes immediately
Result: 3 separate full-data passes
PySpark (Lazy Evaluation):
──────────────────────────
df.filter(...) ← builds plan node, NO execution
df.groupBy(...) ← adds to plan, NO execution
df.select(...) ← adds to plan, NO execution
│
│ .write() / .show() / .collect() called
▼
Spark Optimizer analyzes the full plan:
- Pushes filter early (less data to scan)
- Combines operations into single data pass
- Chooses optimal join strategy
- Decides partition count per stage
│
▼
Optimal execution across cluster — ONCE
Pandas vs PySpark — how to decide
There is no "always use X" rule here. Data size and available infrastructure decide it together.
- ✓ Data size < 1–2 GB (it fits in RAM)
- ✓ The work runs on a single machine
- ✓ Fast prototyping / exploration
- ✓ Local development, notebooks
- ✓ ML preprocessing (alongside sklearn)
- ✗ GB+ files — will crash with MemoryError
- ✗ Production pipelines at scale
- ✓ Data size > 5–10 GB (you need a cluster)
- ✓ Distributed infrastructure is available
- ✓ Parallel processing is required
- ✓ Production data pipelines (Databricks, EMR)
- ✓ Streaming data (Spark Streaming)
- ✗ Small data — overhead without benefit
- ✗ Quick exploration — slower to iterate
| Scenario | Data Size | Tool | Why |
|---|---|---|---|
| Startup analytics dashboard | 100 MB | Pandas | Single machine, fast iterations, simple setup |
| Daily ETL — small business | 500 MB | Pandas | Fits in RAM, no cluster overhead needed |
| E-commerce order history | 5–20 GB | PySpark | Exceeds typical RAM, needs partitioning |
| Swiggy / Zomato daily logs | 100+ GB | PySpark | Only distributed processing can handle this |
| IRCTC booking transactions | TB-scale | PySpark | Petabyte-capable — only option at this scale |
- Python is the default language for data engineering because the whole ecosystem — Pandas, PySpark, Airflow, dbt, Luigi, Great Expectations — is built in it. No other language comes close to that level of maturity.
- SQL is powerful but limited: it queries, it does not build pipelines. Multi-source merging, raw data cleaning, scheduling and error handling are all Python code.
- Java and Scala are technically capable, but ecosystem gravity pulls toward Python. New data engineering tools are built Python-first, and tooling in Scala or Go is sparse.
- Pandas loads the entire dataset into one machine's RAM and processes it sequentially on a single core. On multi-gigabyte data it crashes with
MemoryError. - PySpark splits data into partitions, distributes them across cluster nodes, and processes them in parallel. Lazy evaluation optimizes the full execution plan first, then executes it in the fewest possible data passes.
- The decision rule: does the data fit in RAM? Use Pandas. Does it need a cluster? Use PySpark. Data size and infrastructure pick the tool, not personal preference.
Want the data engineering roadmap?
Save the reel so you can revise later.
Comment "DATA" on Instagram and the full data engineering roadmap lands in your DMs.
Don't miss the rest of the series.