{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# PySpark Tutorial for Data Science Beginners\n",
        "Xenors practice notebook. Run each cell and compare the output with the tutorial page.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Inspect partitions before tuning\n",
        "Partition count affects parallelism; tune it after measuring stage time and shuffle size.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pyspark.sql import SparkSession\n",
        "spark = SparkSession.builder.getOrCreate()\n",
        "df = spark.range(0, 1_000_000).repartition(8)\n",
        "print(df.rdd.getNumPartitions())\n",
        "print(df.groupBy().count().collect()[0][0])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Expected/checkpoint**\n",
        "```text\n8\n1000000\n```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Compare cached and uncached actions\n",
        "Cache only reused data; caching everything can waste executor memory.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df2 = df.withColumn('bucket', (df.id % 10))\n",
        "df2.cache()\n",
        "df2.count()\n",
        "print(df2.groupBy('bucket').count().orderBy('bucket').show(3))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Expected/checkpoint**\n",
        "```text\nbucket | count\n0      | 100000\n1      | 100000\n2      | 100000\n```\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.x"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}