XenorsPySpark Tutorial: Big Data Processing in Python
When data no longer fits in one machine's memory, pandas stops being enough. Spark splits data into partitions and processes them in parallel across a cluster, and PySpark is its Python API.
How Spark works
A driver program builds a plan and executors run tasks on partitions of data. Transformations such as select or filter are lazy and only describe the plan. Actions such as count, show or write run it. Wide transformations like groupBy or join require a shuffle, which moves data across the network and is the main cost to watch.
| Use pandas when | Use PySpark when |
|---|---|
| Data fits in memory | Data is larger than one machine |
| Quick exploration | Distributed ETL and pipelines |
| Single machine | Cluster or cloud (Databricks, EMR) |
Start Spark and read data
Local mode lets you learn on a laptop with the same API used on a cluster.
# pip install pyspark
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.appName("demo").master("local[*]")
.config("spark.sql.shuffle.partitions", "8").getOrCreate())
df = spark.read.csv("sales.csv", header=True, inferSchema=True)
df.printSchema()
df.show(5)
print("rows:", df.count())Transform and aggregate
Prefer built-in functions in pyspark.sql.functions over Python UDFs, because they run inside the optimised engine.
result = (df.filter(F.col("amount") > 0)
.withColumn("month", F.date_format("order_date", "yyyy-MM"))
.groupBy("month", "region")
.agg(F.sum("amount").alias("revenue"),
F.countDistinct("customer_id").alias("customers"))
.orderBy("month", F.desc("revenue")))
result.show(10)
result.write.mode("overwrite").partitionBy("month").parquet("out/revenue")Train a model with MLlib
MLlib models expect one vector column of features.
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.evaluation import BinaryClassificationEvaluator
data = df.select("age", "income", "visits", F.col("churned").cast("int").alias("label"))
vec = VectorAssembler(inputCols=["age", "income", "visits"], outputCol="features").transform(data)
train, test = vec.randomSplit([0.8, 0.2], seed=42)
model = LogisticRegression(featuresCol="features", labelCol="label").fit(train)
auc = BinaryClassificationEvaluator(labelCol="label").evaluate(model.transform(test))
print("AUC:", round(auc, 3))Performance habits
Use Parquet instead of CSV for repeated work. Filter early so less data is shuffled. Cache a DataFrame only when you reuse it several times. Avoid collect() on large data, because it pulls everything into the driver. Inspect plans with df.explain() and the Spark UI at port 4040.
Hands-on practice
Run the examples in the downloadable Jupyter notebook. The extra checks below focus on mistakes that appear in real projects, not just syntax.
Inspect partitions before tuning
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.range(0, 1_000_000).repartition(8)
print(df.rdd.getNumPartitions())
print(df.groupBy().count().collect()[0][0])8 1000000
Partition count affects parallelism; tune it after measuring stage time and shuffle size.
Compare cached and uncached actions
df2 = df.withColumn('bucket', (df.id % 10))
df2.cache()
df2.count()
print(df2.groupBy('bucket').count().orderBy('bucket').show(3))bucket | count 0 | 100000 1 | 100000 2 | 100000
Cache only reused data; caching everything can waste executor memory.
Self-check
- Can you explain why the first practice result is trustworthy rather than just plausible?
- What would you monitor or test before using this PySpark Fundamentals workflow in production?
- Which assumption in the example would break first on a larger or messier dataset?
Frequently asked questions
Is PySpark faster than pandas?
On small data pandas is usually faster because Spark has startup overhead. PySpark wins when data or computation must be distributed.
What is lazy evaluation?
Spark records transformations as a plan and only executes when an action is called, which lets it optimise the whole plan.
Do I need a cluster to learn PySpark?
No. Local mode runs the full API on your laptop.
Primary references
Use these official sources when exact syntax, defaults or definitions matter beyond the examples here.
