A fitted model is half the job. The other half is deciding whether it is any good — and "good" depends entirely on which kind of wrong you can live with. Every metric below is a different answer to the same question: how far off were we, and in which direction?
Five sections, five definitions of error. We start with R², the fraction of variance a regression line explains. Then MSE / RMSE / MAE, which score residuals — and disagree loudly about outliers. We switch to classification: accuracy first (and why it lies), then precision / recall / F1 (which kinds of wrong you are doing), and finally ROC / AUC — one curve that summarises every threshold at once.
Each section has a graph you can poke at. The readouts update live; the goal is to build an intuition for how each metric moves when the data moves.
R² answers a single question: of all the spread in y, how much did our line account for? The baseline is the flat mean — a model that predicts \(\bar{y}\) no matter the input. That baseline has R² = 0. A perfect fit has R² = 1. Drag the noise slider to swell the cloud and watch R² drop toward the mean line.
0 = perfect line (R² = 1). 3 = heavy scatter (R² → 0).
R²
…
SStot (variance of y): …
SSres (leftover after fit): …
…
Blue points: data. Solid blue: OLS fit. Dashed: mean of y (R² = 0 baseline).
R² is relative. Often you want an absolute error in the same units as the target. Three classic choices, and they disagree about outliers:
MSE squares the residuals, so one large error can dominate. MAE only takes absolute values, so it grows linearly. Grab the ringed outlier and drag it away from the line — watch MSE rocket past MAE.
Drag the ringed point. Rose bars = above the line; emerald = below.
…
Classification starts with the obvious metric: the fraction of predictions you got right. Slide the threshold across the two score distributions and watch the confusion matrix fill in. But notice what accuracy hides — it cannot tell you which errors you are making, only how many.
Predict positive iff score ≥ threshold.
Confusion matrix
Accuracy: …
…
Blue = negatives, green = positives. Dashed rose = threshold.
Accuracy lumps every error together. Two questions untangle them:
Precision asks: of the points we flagged positive, how many really were? Recall asks: of the real positives, how many did we catch? They pull against each other — raise the threshold and precision climbs while recall falls. The rug plot below colours each sample by its role: emerald TP, rose FP, amber FN, faded slate TN.
…
Upper rug = positives, lower rug = negatives. Each tick is one sample.
Every threshold gives you one (FPR, TPR) point. Sweep the threshold from high to low and those points trace the ROC curve. The area under it — AUC — is a single number that summarises how well the score separates the classes, independent of any threshold. Slide the distributions apart and watch the curve crawl toward the top-left corner; AUC climbs toward 1.
Negative shift overlaps the classes (AUC → 0.5); positive shift separates them (AUC → 1).
Score distributions + current threshold.
ROC curve in (FPR, TPR) space. Diagonal = random.
…
These five metrics are the vocabulary every later essay takes for granted. R² is what cross-validation reports when we tune Ridge and Lasso on the breach dataset. MSE (and its gradient) is literally the loss surface gradient descent walks down in the calculus primer. AUC is how we compare two bandit policies when neither dominates the other at every threshold.
Two practical warnings to carry forward. First, no single metric is a verdict. A model with 95% accuracy and 30% recall on the positive class is a model that misses most of what you care about — accuracy alone would have hidden that. Read precision and recall and the confusion matrix; reach for ROC/AUC when you want a threshold-free summary.
Second, always look at the distribution of errors, not just their average. MSE folds everything into one number; the residual plot in Section 2 is the diagnostic that tells you why MSE is what it is. Outliers, heteroscedasticity, a missed nonlinearity — none of them show up in the number, all of them show up in the plot. Train yourself to reach for the plot first and the number second.