Statistical MeasuresThe Numbers That Describe Numbers

A dataset is too big to look at. Even seven exam scores are hard to reason about as a list; a real dataset with a million rows is impossible. So statistics offers a deal: give up the details, keep the essence. A statistical measure is a single number that summarizes one property of the whole collection.

The properties worth summarizing come in four families, and this note walks through all of them:

This is exactly what df.describe() prints when a machine learning project begins. Every choice that follows, from how to fill missing values to how to scale features and which loss function to train with, leans on these numbers. Understanding what they mean is the difference between running that command and reading it.

One small dataset will follow us through the whole note. Seven students took a quiz out of 100:

Small enough to compute everything by hand, real enough to show every idea.

The arithmetic mean is the total shared out equally: add everything, divide by the count. For values \(x_1, x_2, \ldots, x_n\):

For our quiz scores:

The formula hides a lovely physical picture. Put the values as equal weights on a ruler. The mean is where the ruler balances:

Two properties make the mean the workhorse of all statistics. First, it uses every value; nothing is wasted. Second, it is the unique number that minimizes the sum of squared distances to the data: among all candidates \(c\), the quantity \( \sum (x_i - c)^2 \) is smallest exactly when \(c = \bar{x}\). The mean is the best single guess if being wrong is punished quadratically.

But "uses every value" is also its weakness. Every point pulls on the fulcrum, so one wild value drags the mean toward itself. We will watch this happen live in a moment.

Sort the values and take the one in the middle. That is the median: the value with half the data below it and half above.

With an odd count \(n\), the median is the \(\frac{n+1}{2}\)-th sorted value. With an even count there is no single middle, so we average the two middle values:

Notice what the median ignores: everything except order. It doesn't care whether the top score is 70 or 700; that student is simply "above the middle" either way. This blindness is a feature. Statisticians call it robustness: resistance to outliers.

The median has its own optimality story, a mirror of the mean's. The median minimizes the sum of absolute distances \( \sum |x_i - c| \). Punish errors by squared distance and the best guess is the mean; punish by plain distance and the best guess is the median.

Don't take robustness on faith. The figure below shows our seven scores as dots. The last one is yours. Drag it to the right and watch which marker chases it:

This is the whole mean-vs-median decision in one picture. When data is clean and symmetric they agree, and the mean's efficiency wins. When data has heavy tails or errors, as with incomes, house prices, hospital stays, and web-page load times, the median tells the truer story of the "typical" case.

The mode is simply the value that occurs most often. In our quiz scores, 50 appears twice and everything else once, so the mode is 50. It is the only measure of center that works for categorical data: the mode of {red, blue, blue, green} is blue, and no mean or median even makes sense there.

For continuous data the mode is better read from a histogram: it is the tallest bar, the peak of the distribution. And that reading reveals something the mean and median both hide: a distribution can have two peaks.

Center is half the story. Two classes can both average 50 while one is packed between 45 and 55 and the other sprawls from 0 to 100. The second family of measures asks: how spread out is the data?

The crudest answer is the range, maximum minus minimum: \(70 - 35 = 35\) for our quiz. Quick, but it depends on exactly the two most extreme values, the least trustworthy points in any dataset. One typo and the range explodes.

A sturdier idea: instead of the two endpoints, use landmarks inside the data. The p-th percentile is the value below which \(p\%\) of the data falls. Three percentiles are so useful they get their own names, the quartiles:

For the quiz scores, split the sorted list at the median and take the median of each half: lower half \(\{35, 40, 45\}\) gives \(Q_1 = 40\); upper half \(\{50, 60, 70\}\) gives \(Q_3 = 60\). The interquartile range is the width of the middle half of the data:

These five numbers (min, \(Q_1\), median, \(Q_3\), max) are the five-number summary, and the box plot is that summary drawn to scale:

The IQR also gives statistics its standard outlier fence. Any point below \(Q_1 - 1.5 \times \text{IQR}\) or above \(Q_3 + 1.5 \times \text{IQR}\) is flagged as an outlier. For the quiz: fences at \(40 - 30 = 10\) and \(60 + 30 = 90\). All seven scores sit comfortably inside. But the dragged point from figure 02 at 250? Far beyond the fence. The rule catches exactly the kind of value that hijacked the mean.

The IQR measures spread with two landmarks. Variance measures it with every point, the same trade the mean makes for center. Start with the natural idea: how far is each value from the mean?

Average these and you get exactly zero. The fulcrum property from figure 01 guarantees the negatives cancel the positives, always, for any dataset. The fix is to make every deviation positive before averaging, and squaring is the mathematically friendly way to do it. The variance is the mean of the squared deviations:

By hand, for the quiz:

But 121.4 what? Squaring the deviations squared the units too, so this is 121.4 marks². Nobody thinks in squared marks. So we take the square root to come back to the original scale, and that is the standard deviation:

Read it as: a typical student sits about 11 marks away from the average. Variance is the machinery (it adds nicely, it differentiates nicely, so the math of statistics runs on it); standard deviation is the human-readable readout of the same information.

One honest wrinkle. If your seven values are a sample from a bigger population (seven students standing in for the whole university), dividing by \(n\) systematically underestimates the population's spread, because a sample rarely captures the extremes. The correction is to divide by \(n - 1\) instead:

This is Bessel's correction. The intuition: the deviations are measured from \(\bar{x}\), which was itself computed from the same data, so the deviations are slightly too small on average; one "degree of freedom" was spent estimating the mean. NumPy's np.std() divides by \(n\) by default; pandas' .std() divides by \(n-1\). Countless silent bugs live in that difference.

To feel what σ does to a distribution's shape, pull the slider:

Center and spread still miss one thing: symmetry. Quiz scores cluster symmetrically; incomes do not: most people earn modest amounts, a few earn fortunes, and the histogram grows a long tail to the right. Skewness measures that lopsidedness, and its sign tells you which side the tail is on.

This is why "average income" is a misleading phrase. The long right tail of a few billionaires pulls the mean well above what a typical person earns, which is why economists report median income instead. Same data, different measure, different story.

Here the measures stop being separate tools and start working together. Take any value, subtract the mean, divide by the standard deviation:

The student who scored 70, in a class with mean 50 and sample std 11.9:

"1.7 standard deviations above average." The magic is that this sentence is unit-free and dataset-free. A quiz score of 70, a height of 190 cm, a salary: once converted to z-scores, they all live on the same ruler and can be compared. Is a 70 on this quiz more impressive than 180 cm in height? Compare the z-scores; the raw numbers can't be compared, the z-scores can.

The 68–95–99.7 rule from figure 05 turns z-scores into a surprise meter: for bell-shaped data, \(|z| > 2\) puts a value in the rarest 5%, and \(|z| > 3\) in the rarest 0.3%, the classic statistical definition of an outlier.

Apply the same recipe to an entire feature column and you get standardization:

Every standardized feature is centered at zero with unit spread. Its cousin, min-max normalization \( x' = \frac{x - \min}{\max - \min} \), squeezes values into [0, 1] instead. It is simpler, but built from the fragile min and max, so one outlier crushes all the other values into a tiny interval. Standardization inherits some of the mean's outlier-sensitivity too, which is exactly why the median/IQR-based robust scaler from section 05 exists. Every scaling method is just a choice of which center and which spread to trust.

Everything so far described one column at a time. The last family asks about pairs: when study hours go up, do scores go up too? Covariance answers by multiplying paired deviations:

Read the product's sign. If a student is above average in hours and above average in score, both deviations are positive and the product is positive. Below average in both: two negatives, still positive. Mismatched: negative. So the sum comes out positive when the variables move together, negative when they move oppositely, near zero when they ignore each other.

Covariance has the mean's old disease, though: units. Its value depends on the scales of both variables (hours × marks?), so "cov = 73" is uninterpretable by itself. The cure is also the old one: divide out the spreads, z-score style.

This is Pearson's correlation coefficient. It is unit-free, and its value finally speaks plainly: +1 is a perfect straight-line rise, −1 a perfect fall, 0 no linear pattern at all. Drag the slider through the whole range:

Two warnings, both famous for being ignored. First, r measures straight-line association only: a perfect U-shaped relationship (like \(y = x^2\) on symmetric data) has \(r \approx 0\). Zero correlation does not mean no relationship, so always plot the scatter. Second, correlation is not causation: ice-cream sales and drowning deaths correlate strongly, caused by a third variable, summer. A correlated feature can still be a useful predictor, but only an experiment can establish cause.

Every measure in one table: what it computes, whether an outlier can hijack it, and where it shows up in an ML pipeline:

And the whole note, verified in eight lines of Python:

# every measure in this note, on the running example
import numpy as np
x = np.array([35, 40, 45, 50, 50, 60, 70])

print(x.mean(), np.median(x))               # 50.0  50.0
print(np.percentile(x, [25, 50, 75]))       # [42.5 50. 55.] · see note below
print(x.var(), x.std())                     # 121.43  11.02   (÷ n, population)
print(x.var(ddof=1), x.std(ddof=1))         # 141.67  11.90   (÷ n−1, sample)
print((x - x.mean()) / x.std(ddof=1))       # z-scores · last one ≈ 1.68

One honest footnote to that output: NumPy reports the quartiles as 42.5 and 55, not the 40 and 60 we computed by hand. Both are correct. The hand method (median of each half, "Tukey's hinges") and NumPy's default (linear interpolation between sorted values) are two of several legitimate quartile conventions, and they disagree on small datasets. On a thousand rows the difference vanishes; on seven, it shows. A useful early lesson: even "simple" statistics carry conventions worth checking.

Ten measures, one thread. The mean trusts every point and pays for it; the median and IQR trade information for safety; variance squares its way past cancellation and σ takes the root back to human units; z-scores put everything on one ruler; correlation is that ruler applied to pairs. Descriptive statistics is a small toolbox, but every model you will ever train picks it up first.

• David Diez, Mine Çetinkaya-Rundel, Christopher Barr · OpenIntro Statistics · chapter 2, summarizing data
• Peter Bruce, Andrew Bruce, Peter Gedeck · Practical Statistics for Data Scientists · chapter 1, exploratory data analysis
• scikit-learn documentation · Preprocessing data · StandardScaler, RobustScaler, and friends
• Companion notes in this series: Linear Regression · Logistic Regression · Gradient Descent