Data Cleaning in Python: Handling Missing Values & Outliers

Build the mental model behind Data Cleaning in Python

A customer-age column containing -1, 999 and text values should first be converted safely to numeric, then validated against a plausible range. Invalid values can be set to missing and handled according to the analysis goal.

• Cleaning is not deleting inconvenient rows; it is making data quality decisions explicit and reproducible.
• Missing values can mean unavailable, not applicable or genuinely absent, and those cases should not be treated identically.
• Outlier handling depends on context: a rare but valid transaction is different from a sensor error.
• Keep raw data immutable and record every cleaning rule so another analyst can reproduce the dataset.

Run small examples and inspect every intermediate object

For Data Cleaning in Python, predict one visible result before each snippet runs—a row count, shape, statistic, ranking or error pattern—then compare it with the printed checkpoint.

# Data Cleaning in Python · example 1
import pandas as pd

df['age'] = pd.to_numeric(df['age'], errors='coerce')
df.loc[~df['age'].between(0, 110), 'age'] = pd.NA
df = df.drop_duplicates(subset='customer_id')
print(df['age'].isna().mean())

Data Cleaning in Python checkpoint 1: Run the cell and inspect the printed values before changing anything.

# Data Cleaning in Python · example 2
import pandas as pd

df = pd.read_csv('practice.csv')
print(df.shape)
print(df.dtypes)
print(df.head(3).to_string(index=False))

Data Cleaning in Python checkpoint 2: (30, 5)
Column types are printed, followed by the first three rows.

# Data Cleaning in Python · example 3
missing = df.isna().sum().sort_values(ascending=False)
print(missing[missing.gt(0)])

Data Cleaning in Python checkpoint 3: Only columns with at least one missing value are listed.

# Data Cleaning in Python · example 4
summary = df.select_dtypes('number').agg(['count','mean','median','min','max']).T
print(summary.round(2))

Data Cleaning in Python checkpoint 4: A compact numeric summary appears with one row per numeric column.

# Data Cleaning in Python · example 5
test = df.copy()
if len(test):
    test.iloc[0, 0] = test.iloc[0, 0]
print('rows:', len(test), 'duplicates:', test.duplicated().sum())

Data Cleaning in Python checkpoint 5: The row count and duplicate count make the check visible instead of implicit.

Debug shape, dtype and missing-value problems

Self-check with answers

Turn the lesson into a reusable notebook