{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Descriptive Statistics: Mean, Median, Mode, Variance & Standard Deviation\n",
        "Practice notebook for the Xenors lesson: https://xenors.in/data-science/descriptive-statistics-data-science/\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "df = pd.read_csv('../datasets/descriptive-statistics-data-science.csv')\n",
        "df.head()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Baseline example\n",
        "Expected checkpoint: Run the cell and inspect the printed values before changing anything.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "s = pd.Series([20, 25, 25, 30, 35, 80, 20000])\n",
        "print('mean', s.mean())\n",
        "print('median', s.median())\n",
        "print('std', s.std())\n",
        "print('IQR', s.quantile(.75) - s.quantile(.25))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Summarize the sample\n",
        "Expected checkpoint: n, mean and median are printed from the first numeric column.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "df = pd.read_csv('practice.csv')\n",
        "x = df.select_dtypes('number').iloc[:,0].dropna()\n",
        "print('n=', len(x), 'mean=', round(x.mean(),2), 'median=', round(x.median(),2))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Measure spread\n",
        "Expected checkpoint: Standard deviation and IQR provide two different views of spread.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "q1, q3 = x.quantile([0.25,0.75])\n",
        "iqr = q3 - q1\n",
        "print('std=', round(x.std(ddof=1),2), 'IQR=', round(iqr,2))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Bootstrap one statistic\n",
        "Expected checkpoint: The three values are the bootstrap 2.5th, 50th and 97.5th percentiles.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "rng = np.random.default_rng(42)\n",
        "boot = [rng.choice(x, size=len(x), replace=True).mean() for _ in range(1000)]\n",
        "print(np.percentile(boot,[2.5,50,97.5]).round(2))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Check sensitivity to an extreme value\n",
        "Expected checkpoint: The mean usually moves more than the median when one extreme value is added.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "x2 = pd.concat([x, pd.Series([x.max()*5])], ignore_index=True)\n",
        "print('mean change:', round(x2.mean()-x.mean(),2))\n",
        "print('median change:', round(x2.median()-x.median(),2))\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.x"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}