Grid Search

Definition

Grid search is a hyperparameter tuning method that exhaustively evaluates a model across every combination of hyperparameter values from a predefined discrete set for each parameter, selecting the combination with the best validation performance.

Overview

Machine learning models typically have hyperparameters — settings like learning rate, regularization strength, tree depth, or number of layers — that are not learned from data but must be chosen before training. Grid search is one of the simplest and most widely taught approaches for choosing these values: the practitioner specifies a small, discrete list of candidate values for each hyperparameter, forming a multi-dimensional grid, and the algorithm trains and evaluates a model for every possible combination of values across that grid. For example, tuning a support vector machine might involve a grid of three values for the regularization parameter C and three values for the kernel coefficient gamma, producing nine total combinations to train and evaluate, typically using cross-validation to get a robust estimate of each combination's performance. The combination that achieves the best validation score is selected as the final hyperparameter configuration. Grid search's main advantage is its simplicity and completeness within the specified grid — it is guaranteed to find the best combination among the candidates provided, and it is straightforward to parallelize since each combination can be evaluated independently. Its major drawback is the combinatorial explosion of cost: the number of combinations grows multiplicatively with the number of hyperparameters and the number of candidate values per parameter, making it computationally infeasible for models with many hyperparameters or expensive training costs, such as deep neural networks. Because of this scalability limitation, grid search is most practical for models with few hyperparameters and relatively fast training times, such as classical machine learning models like random forests or support vector machines. For higher-dimensional or more expensive search spaces, alternatives like random search or Bayesian optimization are generally preferred, since they can explore a comparable or larger effective search space with far fewer total evaluations.

Key Concepts

• Exhaustively evaluates every combination of specified hyperparameter values
• Guaranteed to find the best combination within the defined grid
• Cost grows multiplicatively with the number of hyperparameters and values
• Typically combined with cross-validation for robust performance estimates
• Easy to parallelize since each combination is evaluated independently
• Best suited to models with few hyperparameters and fast training
• Simple to implement and interpret compared to more advanced search methods
• Widely available in libraries such as scikit-learn's GridSearchCV

Use Cases

Frequently Asked Questions

Related in AI Fundamentals