Understand precision, recall, F1-score, ROC-AUC, PR-AUC, confusion matrices and threshold selection. Every Model Evaluation example includes a visible checkpoint so you can verify the reasoning instead of trusting the code blindly. This page targets the practical intent behind evaluation measures, not repetition of the phrase itself.
Editorial note: Publication metadata for Model Evaluation is intentionally unset until this URL is actually released; the lesson already includes its own examples, CSV and notebook.
Frame the prediction problem before choosing a model
In fraud detection, a model can have high accuracy by predicting almost everything as legitimate. Precision-recall metrics expose whether the rare fraud cases are actually being found.
- Precision answers how many predicted positives were correct, while recall answers how many actual positives were found.
- F1 combines precision and recall through a harmonic mean, making it useful when both types of error matter.
- ROC-AUC measures ranking across thresholds, but precision-recall curves can be more informative for rare positives.
- Metric choice should reflect the cost of mistakes and the decision threshold used in production.
Build a baseline and a leakage-safe evaluation
For Model Evaluation, predict one visible result before each snippet runs—a row count, shape, statistic, ranking or error pattern—then compare it with the printed checkpoint.
# Model Evaluation · example 1
from sklearn.metrics import classification_report, roc_auc_score
print(classification_report(y_test, y_pred))
print('ROC-AUC:', roc_auc_score(y_test, y_prob))Model Evaluation checkpoint 1: Run the cell and inspect the printed values before changing anything.
# Model Evaluation · example 2
import pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_csv('practice.csv')
target = df.select_dtypes('number').columns[-1]
X = df.drop(columns=[target]).select_dtypes('number')
y = df[target]
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.25,random_state=42)
print(X_train.shape, X_test.shape)Model Evaluation checkpoint 2: Training and test shapes are printed so the split is explicit.
# Model Evaluation · example 3
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error
baseline=DummyRegressor(strategy='mean').fit(X_train,y_train)
pred=baseline.predict(X_test)
print('baseline MAE=', round(mean_absolute_error(y_test,pred),3))Model Evaluation checkpoint 3: The baseline score becomes the minimum standard a more complex model should beat.
# Model Evaluation · example 4
from sklearn.linear_model import LinearRegression
model=LinearRegression().fit(X_train,y_train)
pred=model.predict(X_test)
print('model MAE=', round(mean_absolute_error(y_test,pred),3))Model Evaluation checkpoint 4: Compare model MAE with the baseline instead of judging the model in isolation.
# Model Evaluation · example 5
errors=(y_test-pred).rename('residual')
print(errors.describe().round(3))
print('largest absolute errors:')
print(errors.abs().sort_values(ascending=False).head())Model Evaluation checkpoint 5: Residual statistics and the largest errors reveal cases worth investigating.
Inspect errors instead of chasing one score
ROC-AUC measures ranking across thresholds, but precision-recall curves can be more informative for rare positives.
Metric choice should reflect the cost of mistakes and the decision threshold used in production.
Self-check with answers
For Model Evaluation, what result should you be able to explain without the code?
Precision answers how many predicted positives were correct, while recall answers how many actual positives were found.
Which assumption matters most in this Model Evaluation exercise?
ROC-AUC measures ranking across thresholds, but precision-recall curves can be more informative for rare positives.
What failure mode would make this Model Evaluation result untrustworthy?
Metric choice should reflect the cost of mistakes and the decision threshold used in production.
Repeat the experiment with a documented change
Try it yourself workspace
Edit this Model Evaluation starter, copy it, and run it in the downloadable notebook or your local Jupyter environment.
Questions specific to this lesson
What should I establish before tuning evaluation measures?
For Model Evaluation, begin with a leakage-safe split and a simple baseline so later tuning has a credible reference point.
How do I check overfitting in Model Evaluation?
For Model Evaluation, compare training and validation behavior, inspect held-out errors, and repeat the evaluation with a stable split or cross-validation when appropriate.
Which metric should I report for Model Evaluation?
Metric choice should reflect the cost of mistakes and the decision threshold used in production.
How should I explain Model Evaluation in a portfolio or interview?
When presenting Model Evaluation, show the problem definition, baseline, split strategy, metric, one error-analysis finding and one limitation.
Primary references and further reading
For Model Evaluation, use these primary references when exact syntax, defaults or formal definitions matter beyond the examples on this page.
