Logistic Regression Explained
Despite the name, logistic regression is a classification algorithm, not a regression algorithm in the sense of predicting continuous values. It estimates the probability that an observation belongs to a particular class, using a linear combination of features passed through the sigmoid (logistic) function to squash outputs into the range (0, 1). This makes it one of the most widely used baseline classifiers: it is fast to train, produces well-calibrated probability estimates when assumptions hold reasonably well, and its coefficients are directly interpretable in terms of log-odds.
Cricket analogy: Predicting whether India wins a match isn't a continuous score prediction - logistic regression squashes a weighted combination of run rate and wickets in hand through the sigmoid to output something like a 73% win probability, fast to compute and easy to explain to commentators.
The Sigmoid Function and Log-Odds
Logistic regression first computes a linear score z = b0 + b1*x1 + ... + bn*xn, exactly as linear regression would, but then passes z through the sigmoid function sigma(z) = 1 / (1 + e^-z) to produce a probability between 0 and 1. The inverse relationship is that the log-odds (logit) of the positive class is linear in the features: log(p / (1-p)) = z. This means each coefficient bi represents the change in log-odds of the positive class for a one-unit increase in xi — exponentiating a coefficient gives an odds ratio, a common way to communicate effect size to non-technical audiences (e.g., 'each additional year of age multiplies the odds of the outcome by 1.05').
Cricket analogy: Logistic regression computes a linear score z from run rate and wickets, then passes it through sigmoid for a win probability; since log-odds are linear in features, 'each extra wicket in hand multiplies win odds by 1.4' is a natural way to explain it to a commentator.
Training via Maximum Likelihood
Unlike linear regression, logistic regression is not fit by minimizing sum of squared errors — instead it maximizes the likelihood of the observed class labels under the model, equivalently minimizing the log-loss (binary cross-entropy): -mean(y*log(p) + (1-y)*log(1-p)). This loss heavily penalizes confident, wrong predictions (predicting probability near 0 for an actual positive, or near 1 for an actual negative), which encourages well-calibrated probabilities rather than just correct hard classifications. There is no closed-form solution, so the model is fit iteratively (e.g., via gradient descent or Newton's method / L-BFGS as scikit-learn uses by default).
Cricket analogy: Instead of squared error, the model is fit by log-loss, which heavily punishes a confident 95% win prediction that ends in a shock loss, like an easy chase collapsing to 87 all out; there's no shortcut formula, so weights are found iteratively, like gradient descent.
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, log_loss
rng = np.random.default_rng(3)
n = 400
age = rng.uniform(20, 70, n)
income = rng.normal(50000, 15000, n)
z = 0.06 * (age - 45) + 0.00003 * (income - 50000) + rng.normal(0, 1, n)
prob = 1 / (1 + np.exp(-z))
y = (rng.uniform(size=n) < prob).astype(int)
X = np.column_stack([age, income])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=3)
clf = make_pipeline(StandardScaler(), LogisticRegression())
clf.fit(X_train, y_train)
probs = clf.predict_proba(X_test)[:, 1]
preds = clf.predict(X_test)
print("accuracy:", accuracy_score(y_test, preds))
print("log loss:", log_loss(y_test, probs))
print("coefficients (scaled space):", clf.named_steps['logisticregression'].coef_)
# Odds ratio interpretation for the first (standardized) coefficient
coef = clf.named_steps['logisticregression'].coef_[0]
print("odds ratios:", np.exp(coef))A useful intuition: linear regression draws a straight line through the data; logistic regression draws a straight decision boundary through feature space and reports how confidently each point sits on either side of it, expressed as a probability via the sigmoid's S-shaped squashing curve.
The default classification threshold of 0.5 is a convention, not a law — for imbalanced classes or asymmetric costs (e.g., missing a fraud case is far worse than a false alarm), the threshold should be tuned based on precision/recall or cost analysis rather than left at 0.5, since accuracy at threshold 0.5 can be misleadingly high on imbalanced data.
Regularized Logistic Regression
Scikit-learn's LogisticRegression applies L2 regularization by default (controlled by the inverse-strength parameter C, where smaller C means stronger regularization), which helps prevent overfitting and stabilizes coefficients when features are correlated — the same motivation as ridge regression for linear models. L1-penalized logistic regression is also available and, like lasso, can drive some coefficients to exactly zero for implicit feature selection.
Cricket analogy: By default the model applies L2 regularization (like ridge) so a fluky boundary count doesn't dominate the prediction when strike rate and average are correlated; switching to L1 can zero out irrelevant stats entirely, like a scorer's shoe size, and a smaller C tightens this discipline further.
- Logistic regression is a classification algorithm that models class probability, not a continuous target.
- The sigmoid function maps a linear score into a probability between 0 and 1.
- Coefficients represent changes in log-odds; exponentiating gives interpretable odds ratios.
- The model is trained by minimizing log-loss (maximizing likelihood), not sum of squared errors.
- Scikit-learn applies L2 regularization by default via the C hyperparameter (smaller C = stronger regularization).
- The 0.5 decision threshold is a default convention and should be tuned for imbalanced classes or asymmetric error costs.
Practice what you learned
1. What does logistic regression actually predict, despite its name suggesting a regression task?
2. What function does logistic regression use to convert a linear score into a probability?
3. What loss function does logistic regression minimize during training?
4. How should a logistic regression coefficient bi be interpreted after exponentiating it (e^bi)?
5. Why might the default 0.5 classification threshold be inappropriate for a highly imbalanced fraud detection dataset?
Was this page helpful?
You May Also Like
What Is Machine Learning?
An introduction to machine learning as the practice of building systems that improve their performance on a task by learning patterns from data rather than following hand-coded rules.
Confusion Matrix and Classification Metrics
A structured breakdown of correct and incorrect predictions by class, forming the foundation for accuracy, precision, recall, and other classification metrics.
Precision, Recall, and F1 Score
Class-specific metrics that quantify a classifier's trustworthiness on positive predictions (precision) and its ability to find all positives (recall), balanced by the F1 score.
ROC Curves and AUC
A visual and numeric tool for evaluating a binary classifier's ability to distinguish classes across all possible decision thresholds, independent of any single cutoff choice.