Bayesian Optimization
Bayesian optimization is a sample-efficient hyperparameter tuning method that builds a probabilistic surrogate model of how hyperparameters affect performance, using it to intelligently choose the most promising next combination to…
Definition
Bayesian optimization is a sample-efficient hyperparameter tuning method that builds a probabilistic surrogate model of how hyperparameters affect performance, using it to intelligently choose the most promising next combination to evaluate rather than searching blindly.
Overview
Bayesian optimization is designed for tuning expensive-to-evaluate objective functions, such as training a large machine learning model, where each trial can take hours or days and only a limited number of trials are feasible. Unlike grid search or random search, which choose each trial's hyperparameters independently of previous results, Bayesian optimization uses the outcomes of all prior trials to build a probabilistic model — most commonly a Gaussian process, though tree-based surrogates like those used in the Tree-structured Parzen Estimator (TPE) algorithm are also popular — that estimates both the expected performance and the uncertainty of untested hyperparameter combinations. The algorithm alternates between two steps: it fits the surrogate model to all observations gathered so far, and it uses an acquisition function (such as expected improvement or upper confidence bound) to decide which hyperparameter combination to try next. The acquisition function balances exploitation (trying combinations near currently known good regions) against exploration (trying combinations in uncertain regions that might turn out to be even better), which is what makes the search far more sample-efficient than random or grid search. Because it requires fitting and updating a surrogate model at each step, Bayesian optimization has more computational and implementation overhead per trial than random search, but this overhead is negligible compared to the cost of training the underlying model itself in most real-world scenarios, making the tradeoff favorable. It performs best in moderate-dimensional search spaces (roughly up to a few dozen hyperparameters) and struggles to scale efficiently to very high-dimensional spaces. Bayesian optimization is implemented in popular libraries such as Optuna, Hyperopt, and scikit-optimize, and it is widely used for tuning deep learning training recipes, gradient boosting models, and any pipeline where each evaluation is costly enough that minimizing the number of trials matters more than the simplicity of the search strategy.
Key Concepts
- Builds a probabilistic surrogate model of the hyperparameter-to-performance mapping
- Uses an acquisition function to balance exploration versus exploitation
- Far more sample-efficient than grid or random search for expensive evaluations
- Common surrogate models include Gaussian processes and tree-structured Parzen estimators
- Learns from every prior trial rather than sampling combinations independently
- Scales well to moderate-dimensional search spaces, less well to very high dimensions
- Widely implemented in libraries like Optuna, Hyperopt, and scikit-optimize
- Adds per-trial computational overhead that is usually negligible next to training cost
Use Cases
Frequently Asked Questions
From the Blog
LLM Inference Optimization: Latency, Throughput and Cost
Latency, throughput and cost pull against each other in LLM serving, and most optimisation advice fails because it ignores which one you are actually optimising. This guide separates the three goals, maps each technique to the goal it moves, and names the quality or memory price each one charges.
Read More Career GrowthLinkedIn Optimization for Tech Professionals
Optimize your LinkedIn profile to attract recruiters and peers: a clear headline, keyword-rich sections, proof of work, and steady activity that gets you found.
Read More Cloud & CybersecurityCloud Cost Optimization: Where the Bill Actually Comes From
A cloud bill breaks into four families of charge: compute, storage, data transfer and managed services. Optimising without knowing which family dominates wastes effort on the wrong line. This guide shows how to decompose a bill, which lever fits each family, and how to make the reductions stick rather than regrow within a quarter.
Read More