Random Search

Definition

Random search is a hyperparameter tuning method that samples a fixed number of hyperparameter combinations randomly from specified distributions or ranges, rather than exhaustively testing every combination, and selects the best-performing one on a validation set.

Overview

Random search was introduced as a more efficient alternative to grid search for hyperparameter tuning. Instead of defining a small discrete grid and testing every combination, random search defines a distribution or range for each hyperparameter — for example, a log-uniform distribution over learning rate values from 1e-5 to 1e-1 — and independently draws a fixed number of random combinations to train and evaluate. The combination achieving the best validation performance is selected. The key insight behind random search's effectiveness, formalized in influential research, is that in most real-world hyperparameter tuning problems, only a small number of hyperparameters actually have a strong effect on model performance for a given dataset, while others matter much less. Grid search wastes evaluation budget by testing every combination of both important and unimportant hyperparameters, effectively under-sampling the important ones. Random search, by contrast, samples each hyperparameter independently across its full range on every trial, giving much denser coverage of the important dimensions for the same total number of trials. Random search also has a practical advantage in flexibility: unlike grid search, which requires a fixed, symmetric grid, the number of trials in random search can simply be increased or decreased based on available compute budget, and continuous hyperparameters can be sampled from any distribution rather than being restricted to a small discrete set of values. Random search is straightforward to implement and, like grid search, is trivially parallelizable since trials are independent. It generally serves as a strong, simple baseline before moving to more sophisticated methods like Bayesian optimization, which use the results of previous trials to intelligently choose the next combination to try, rather than sampling blindly. Many practitioners use random search as the default first approach precisely because it consistently outperforms grid search under an equal compute budget.

Key Concepts

• Samples a fixed number of hyperparameter combinations at random
• Uses distributions or ranges rather than a small fixed discrete grid
• More efficient than grid search when only a few hyperparameters matter most
• Trial budget is flexible and independent of the number of hyperparameters
• Supports continuous hyperparameter sampling, not just discrete values
• Trivially parallelizable since each trial is independent
• Serves as a strong baseline before Bayesian or other adaptive search methods
• Available in libraries such as scikit-learn's RandomizedSearchCV

Use Cases

Frequently Asked Questions

Related in AI Fundamentals