100% Free Forever
AI-Powered Learning
Industry Expert Content
Certificates & Badges
Learn At Your Own Pace
Machine Learning with Scikit-learn
35 minintermediate

Hyperparameter Tuning with Optuna

Hyperparameter tuning is the process of finding the model configuration — learning rate, tree depth, regularisation strength, number of layers — that produces the best generalisation performance on held-out data. Every machine learning model has hyperparameters that are not learned from data but must be set before training, and their values have a large impact on performance. Choosing them well separates a mediocre model from a competitive one.

Manual tuning is unreliable and inefficient: the space of reasonable hyperparameter combinations is high-dimensional, the relationships between parameters are non-linear and interactive, and human intuition is a poor guide in this space. Grid search is exhaustive but scales exponentially with the number of parameters. Random search is better but still wastes evaluations on clearly poor configurations. Bayesian optimisation — the approach used by Optuna — builds a probabilistic model of the performance landscape and directs evaluations toward promising regions, finding good configurations with far fewer model fits than grid or random search. Optuna is the leading Python library for Bayesian hyperparameter optimisation, combining algorithmic sophistication with a clean, flexible API.

Analogy🏏Cricket
🏏 Think of it like cricket: A batting coach who teaches only one shot — the forward defensive — gives advice with high bias: it is consistently wrong for deliveries that demand a drive or a pull, regardless of how much practice the batsman does. A coach who memorises every ball of the batsman's training career gives advice with high variance: he predicts each training ball perfectly but fails completely on new balls from a different bowler, because he learned the noise of that specific bowler rather than the underlying principles. The great coach finds the balance — teaching the core principles that generalise, without over-specifying for the particular training environment. Just as great coaching lies between the extremes, great ML models balance bias and variance.
Lesson 23 of 35
0% complete