Module 11: MLOps & Model DeploymentBy Ashok Kumar Yadav4 min read

ML Model Monitoring and Data Drift Detection

Models degrade quietly. Customer behaviour changes, a sensor is recalibrated, an upstream field changes meaning, and accuracy falls with no error message.

Types of drift

Data drift means input distributions changed. Concept drift means the relationship between inputs and the target changed. Label drift means class frequencies changed. Data drift is detectable immediately; concept drift needs ground-truth labels, which often arrive late.

Detect drift with the KS test and PSI

The two-sample KS test checks whether two samples plausibly come from one distribution. PSI summarises how far binned distributions moved.

import numpy as np
from scipy.stats import ks_2samp

def psi(expected, actual, bins=10):
    edges = np.percentile(expected, np.linspace(0, 100, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return float(np.sum((a - e) * np.log(a / e)))

rng = np.random.default_rng(0)
train = rng.normal(50, 10, 5000)
prod_ok = rng.normal(50, 10, 2000)
prod_bad = rng.normal(58, 12, 2000)

for name, prod in [("stable", prod_ok), ("shifted", prod_bad)]:
    stat, p = ks_2samp(train, prod)
    print(f"{name:8s} PSI={psi(train, prod):.3f}  KS p={p:.4g}")

Alert thresholds

A common rule of thumb for PSI: below 0.1 is stable, 0.1 to 0.25 is a moderate shift worth watching, and above 0.25 is a major shift. Treat these as starting points and tune them on your own data, since very large samples make statistical tests flag tiny harmless differences.

def drift_report(train_df, prod_df, threshold=0.25):
    out = {}
    for col in train_df.select_dtypes("number"):
        v = psi(train_df[col].dropna().values, prod_df[col].dropna().values)
        out[col] = {"psi": round(v, 3), "alert": v > threshold}
    return out

What else to monitor

Track more than drift: input data quality (nulls, ranges, schema changes), prediction distribution, latency and error rate, throughput, and business outcomes. When labels arrive, compute accuracy or F1 on a rolling window and compare with the offline baseline.

• Log every request with model version and a request id
• Store features and predictions for later analysis
• Automate retraining only after a human-reviewed evaluation gate

Open-source tools

Evidently, WhyLogs and Great Expectations provide ready-made drift reports and data validation. Prometheus and Grafana handle operational metrics such as latency.

import numpy as np
def psi(expected, actual, bins=10):
    q=np.quantile(expected,np.linspace(0,1,bins+1)); q[0]-=1e-9; q[-1]+=1e-9
    e=np.histogram(expected,q)[0]/len(expected); a=np.histogram(actual,q)[0]/len(actual)
    e=np.clip(e,1e-6,None); a=np.clip(a,1e-6,None)
    return np.sum((a-e)*np.log(a/e))
print(round(psi(np.random.normal(0,1,5000), np.random.normal(.4,1,5000)),3))

~0.15 to 0.25 depending on sample

signals={'age_psi':0.18,'auc_baseline':0.84,'auc_current':0.81}
print('feature_drift',signals['age_psi']>0.1)
print('performance_drop',signals['auc_baseline']-signals['auc_current']>0.02)

feature_drift True
performance_drop True

Frequently asked questions

Primary references

Use these official sources when exact syntax, defaults or definitions matter beyond the examples here.

• Evidently AI documentation
• SciPy ks_2samp
• Great Expectations documentation