Machine Learning · ML Notebook · July 2026
Logistic RegressionA Straight Line That Learned to Say Yes or No
Keep the linear machinery, squeeze it through one S-shaped function, and a number-predictor becomes a classifier.
Logistic regression is used when the target variable is categorical, most commonly binary, 0 and 1. Predicting disease (yes/no), email spam detection (spam/not spam), credit default (default/no default). Despite the name, it is really logistic classification: regression machinery, classification job.
Why not just reuse linear regression for this? Suppose we want to predict 0 = not spam, 1 = spam, and we fit
\[ \hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots \]The output of this line can be −2.4, 0.3, 0.8, 17... anything. That's the problem, because a probability must live between 0 and 1, and a straight line happily goes below 0 and above 1. Logistic regression solves this with a two-step design: predict a probability first, then convert it into a class. And the tool that pins any number into \((0,1)\) is the sigmoid function.
First, logistic regression computes a linear score exactly like its cousin:
\[ z = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_n x_n \qquad (\text{in matrix form } z = X\beta) \]Then it passes \(z\) through the sigmoid function to map it to a probability:
\[ P = \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad z \in \mathbb{R},\ \ \sigma(z) \in (0, 1) \]The sigmoid is a smooth, S-shaped function that maps any real-valued input into the range \((0,1)\). Its whole personality in one table:
| z | σ(z) |
|---|---|
| −6 | 0.00247 |
| −3 | 0.04743 |
| −2 | 0.11920 |
| −1 | 0.26894 |
| 0 | 0.5 |
| 1 | 0.73106 |
| 2 | 0.88080 |
| 3 | 0.95257 |
| 6 | 0.99753 |
Domain \((-\infty, +\infty)\), range \((0, 1)\). Far negative scores flatten to almost 0, far positive scores flatten to almost 1, and \(\sigma(0) = 0.5\) sits exactly at the fence.
In summary, logistic regression is a pipeline: linear score → sigmoid → probability → class. If the model says \(P = 0.82\), it thinks there is an 82% chance the class is 1. The last arrow is the decision rule:
\[ \hat{y} = \begin{cases} 1 & \text{if } P \ge 0.5 \\ 0 & \text{if } P < 0.5 \end{cases} \]Here \(P\) is the probability that \(y = 1\) given \(x\), and \(1 - P\) the probability that \(y = 0\). The threshold 0.5 is only a default. It can be adjusted based on the domain (threshold tuning): a cancer screen would rather lower the bar and catch more positives, a spam filter might raise it to avoid deleting real mail. Try it on the slider above.
The sigmoid is not an arbitrary squashing choice, it falls out of a very natural assumption. Start with odds: a ratio comparing the probability an event will happen to the probability it will not:
\[ \text{odds} = \frac{\text{probability of success}}{\text{probability of failure}} = \frac{P}{1 - P} \]Now take the log. The result is called the log-odds or logit (log base \(e\), the natural log):
\[ \log(\text{odds}) = \log\!\Big(\frac{P}{1-P}\Big) \]Logistic regression makes exactly one assumption here: the features have a linear effect on the log-odds:
\[ \log\!\Big(\frac{P}{1-P}\Big) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots = z \]If we assume the log-odds are linear, the probability is forced to become the sigmoid. Unwrap it:
This is the sigmoid function. So we proved: \( \log\frac{P}{1-P} = z \) if and only if \( P = \sigma(z) \). The full model in one boxed line:
\[ P = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots)}} \]That means: linear model → log-odds → probability. The linear score \(z\) ranges over \((-\infty, +\infty)\), the probability of success over \((0, 1)\).
A cost function measures how wrong the model is on the whole training dataset. Linear regression used Mean Squared Error, \( J(\beta) = \frac{1}{2m}\sum (h_\beta(x_i) - y_i)^2 \). Does logistic regression use MSE too? If we tried, it would be
\[ J(\beta) = \frac{1}{m}\sum_i \big(\sigma(z_i) - y_i\big)^2 = \frac{1}{m}\sum_i \Big(\frac{1}{1 + e^{-z_i}} - y_i\Big)^2 \]but there is a problem. In linear regression, MSE is convex in the weights, a clean parabola or bowl, so gradient descent is guaranteed to find the global minimum. In logistic regression, squashing through the sigmoid before squaring the error breaks the clean quadratic structure. The cost surface becomes non-convex, with multiple local minima, and gradient descent can get stuck in one and miss the global minimum. (Convex vs non-convex, with pictures, is in the gradient descent note.)
So instead of MSE, logistic regression uses log loss.
What should a good classification loss do? If the model gives high probability to the correct answer, the loss should be small. If it gives the correct answer a tiny probability, the loss should be enormous. A first idea, simple loss \(= 1 - P\) when \(y = 1\), barely punishes confident stupidity: being 1% sure of the truth loses 0.99, being 0.0001% sure loses 0.999999. Not much difference, yet the second model is ten thousand times more wrong.
The function \(-\log(P)\) fixes exactly this (why negative: for \(0 < x < 1\), \(\log x\) is negative, and the minus flips maximization of probability into minimization of loss; \(\log(1) = 0\) and \(\ln(0)\) is undefined, the loss blows up):
| model's P for the true class | log loss −ln(P) |
|---|---|
| 1.0 | 0 |
| 0.9 | 0.105 |
| 0.5 | 0.693 |
| 0.1 | 2.303 |
| 0.01 | 4.605 |
| 0.001 | 6.908 |
| 0.000001 | 13.816 |
High probability to the correct answer → small loss. Low probability to the correct answer → very large loss. When the actual value is \(y = 1\) the correct probability is \(P\), so the loss is \(-\log(P)\). When \(y = 0\) the correct probability is \(1 - P\), so the loss is \(-\log(1 - P)\):
\[ \text{cost} = \begin{cases} -\log(P) & y = 1 \\ -\log(1 - P) & y = 0 \end{cases} \]The blue curve is \(-\ln(P)\), the penalty when the true class is 1. The teal curve is \(-\ln(1-P)\), the penalty when it is 0. Wherever you place your confidence, one curve forgives you and the other bills you.
The two branches compress into a single line, because \(y\) is always 0 or 1, so one term always vanishes. This is the binary cross-entropy, the general formula:
\[ \text{cost} = -\big[\, y \log(P) + (1 - y)\log(1 - P) \,\big] \]Check it: if \(y = 1\), the loss is \(-[1 \cdot \log P + 0] = -\log(P)\). If \(y = 0\), it is \(-[0 + (1-0)\log(1-P)] = -\log(1-P)\). Averaged over the dataset, this cost is convex for logistic regression, so gradient descent gets its guarantee back. It also has a second name to remember: it prevents positive and negative cancellation, combines all errors, and penalizes big mistakes, everything MSE did for regression, rebuilt for probabilities.
How logistic regression learns: training starts with random coefficients, then it checks the error and changes the coefficients step by step so that the loss becomes smaller. The update formula is the usual gradient descent, \( \beta := \beta - \text{learning rate} \times \text{gradient} \), where the gradient is the direction of increasing loss. So we need \( \frac{\partial(\text{Loss})}{\partial(\text{coefficient})} \).
Here the chain rule earns its keep. When one variable depends on another, which depends on another, derivatives multiply: for \( y = f(g(x)) \), \( \frac{dy}{dx} = \frac{dy}{dg} \times \frac{dg}{dx} \). In logistic regression, the loss depends on the prediction \(P\), the prediction depends on \(z\), and \(z\) depends on the weights:
\[ \frac{\partial L}{\partial \beta} = \frac{\partial L}{\partial P} \times \frac{\partial P}{\partial z} \times \frac{\partial z}{\partial \beta} \]Piece 1. Differentiate \( L = -[y \log P + (1-y)\log(1-P)] \) with respect to \(P\), treating every other variable as a constant:
\[ \frac{\partial L}{\partial P} = -\frac{y}{P} + \frac{1-y}{1-P} \]Piece 2. The sigmoid's own derivative, a small classic. Write \( P = (1 + e^{-z})^{-1} \):
Piece 3. Since \( z = \beta_0 + \beta_1 x_1 + \cdots + \beta_n x_n \): \( \frac{\partial z}{\partial \beta_0} = 1 \), \( \frac{\partial z}{\partial \beta_1} = x_1 \), \( \frac{\partial z}{\partial \beta_2} = x_2 \), in vector form \( \frac{\partial z}{\partial \beta} = x \).
Multiply the three pieces and watch everything cancel:
Gradient = (predicted probability − actual value) × feature value. So the update, summed over the data, is
\[ \beta_j := \beta_j - \alpha \sum_i \big(P_i - y_i\big)\, x_{ij} \]Look closely: this is exactly the same form as the linear regression gradient \((\hat{y} - y)\,x_j\). Logistic regression just computes its prediction through the sigmoid first. Two different losses, two different models, one identical downhill step. That is one of the quiet beautiful facts of ML.
In linear regression, the relationship is direct: with \( \hat{y} = \beta_0 + \beta_1 x \), setting \(\beta_1 = 5\) gives \( \hat{y} = 5x \), doubling it to 10 doubles the effect. A change in a coefficient causes a proportional, linear change in the output.
In logistic regression the coefficient acts through the sigmoid, \( P = \sigma(\beta_0 + \beta_1 x) \). At \(x = 1\) and \(\beta_0 = 0\):
| β₁ | P |
|---|---|
| −5 | 0.0067 |
| 0 | 0.5 |
| +5 | 0.9933 |
So the sign carries the story: \(\beta > 0\) increases the probability of class 1 as the feature grows, \(\beta < 0\) decreases it. The size matters on the log-odds scale, not the probability scale: one unit of \(x_j\) adds \(\beta_j\) to the log-odds, which multiplies the odds by \(e^{\beta_j}\). Near the fence (\(P \approx 0.5\)) that shifts the probability a lot; out at the flat tails, barely at all.
Shorter list than linear regression, and two famous items are missing on purpose.
1 · Independent observations
One row should not depend on another row. Same idea and same failure cases as in linear regression: repeated measurements, time series, clustered groups.
2 · Linear relationship with the log-odds
Logistic regression doesn't assume a linear relationship between the features and the probability. It assumes a linear relationship between the features and the log-odds. The probability curve against a feature is allowed to be S-shaped, that is the whole design.
3 · No multicollinearity
Features should not be highly correlated with each other, or the coefficients become unstable and uninterpretable, exactly as in linear regression.
4 · No extreme outliers
Extreme feature values can dominate the linear score and distort the fit.
5 · No perfect separation
Perfect separation means one feature or combination of features can perfectly separate the classes: if salary > 50,000, always class 1; if salary ≤ 50,000, always class 0. Sounds ideal, but then the likelihood keeps improving as the coefficient grows without bound, the optimizer pushes \(\beta \to \infty\), and the model never settles.
And the missing items: unlike linear regression, logistic regression doesn't require homoscedasticity, and it doesn't require normality of residuals. There is no continuous residual in the same sense, so those assumptions simply don't apply.
Binary logistic regression, used when the target has two classes: yes/no, spam/not spam. Everything in this note.
Multinomial logistic regression, used when the target has more than two classes: predict an animal, cat, dog, bird. This usually uses softmax, the sigmoid's many-class generalization, which turns a vector of scores into a vector of probabilities summing to 1.
Accuracy alone can lie (predict "no disease" always on a dataset that is 99% healthy and score 99%). The standard toolkit starts with the confusion matrix for binary classification:
| predicted 0 | predicted 1 | |
|---|---|---|
| actual 0 | TN | FP |
| actual 1 | FN | TP |
True positives and true negatives are correctly predicted, the other two cells are the two distinct ways of being wrong. From these four numbers:
Accuracy, overall correct predictions. Precision, of the predicted positives, how many were actually positive. Recall, of the actual positives, how many were found. F1-score, the balance (harmonic mean) between precision and recall.
ROC-AUC. The ROC curve (Receiver Operating Characteristic) plots, across all possible thresholds, the false positive rate on the x-axis against the true positive rate on the y-axis:
\[ \text{FPR} = \frac{FP}{FP + TN}, \qquad \text{TPR} = \frac{TP}{TP + FN} \]AUC is the area under that curve, a value between 0 and 1:
| AUC | meaning |
|---|---|
| 1.0 | perfect classifier |
| 0.5 | random guessing |
| < 0.5 | worse than random |
Like linear regression, logistic regression can overfit, and the same three medicines apply, added as penalties to the log loss: L2 regularization (Ridge-style, sum of squared coefficients), L1 regularization (Lasso-style, sum of absolute values, can zero out features), and ElasticNet (both together). The intercept is conventionally left unpenalized.
One scikit-learn quirk worth knowing: for Ridge and Lasso the strength knob is called alpha (bigger = stronger penalty), but LogisticRegression uses C, the inverse of regularization strength, so smaller C means stronger regularization.
# logistic regression from scratch: sigmoid + gradient descent import numpy as np X = np.array([[0.5],[1.0],[1.5],[3.0],[3.5],[4.0]]) # hours of study y = np.array([0, 0, 0, 1, 1, 1]) # pass / fail Xb = np.c_[np.ones(len(X)), X] # add the x0 = 1 column beta = np.zeros(2) for step in range(5000): p = 1 / (1 + np.exp(-(Xb @ beta))) # sigmoid(z) beta -= 0.1 * Xb.T @ (p - y) / len(y) # β := β − α·(P−y)·x print(beta.round(2)) # fence sits near x ≈ 2.25 # the same with scikit-learn from sklearn.linear_model import LogisticRegression clf = LogisticRegression(C=1.0).fit(X, y) print(clf.predict([[2.0],[3.2]]), clf.predict_proba([[3.2]]))
- Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani · An Introduction to Statistical Learning · chapter 4, Classification
- Andrew Ng · CS229 Lecture Notes, Stanford · Part II, Classification and logistic regression
- David Hosmer, Stanley Lemeshow · Applied Logistic Regression · odds, logit, and interpretation
- scikit-learn documentation · LogisticRegression and model evaluation metrics
- Companion notes in this series: Linear Regression · Gradient Descent