Hyperparameter Tuning Cheat Sheet

Strategies and code patterns for searching hyperparameter space, including grid search, random search, and Bayesian optimization with scikit-learn and Optuna.

2 PagesIntermediateMar 18, 2026

Grid & Random Search

Exhaustive and sampled hyperparameter search.

python

from sklearn.model_selection import GridSearchCV, RandomizedSearchCVfrom sklearn.ensemble import RandomForestClassifierfrom scipy.stats import randintparam_grid = {    "n_estimators": [100, 200, 300],    "max_depth": [None, 10, 20, 30],    "min_samples_split": [2, 5, 10],}grid = GridSearchCV(RandomForestClassifier(), param_grid,                     cv=5, scoring="f1", n_jobs=-1)grid.fit(X_train, y_train)print(grid.best_params_, grid.best_score_)# Random search samples a fixed number of combinations -- scales betterparam_dist = {    "n_estimators": randint(50, 500),    "max_depth": randint(3, 50),}random_search = RandomizedSearchCV(RandomForestClassifier(), param_dist,                                    n_iter=50, cv=5, scoring="f1",                                    n_jobs=-1, random_state=42)random_search.fit(X_train, y_train)

Bayesian Optimization with Optuna

Sample promising trials using a probabilistic model.

python

import optunafrom sklearn.model_selection import cross_val_scorefrom sklearn.ensemble import GradientBoostingClassifierdef objective(trial):    n_estimators = trial.suggest_int("n_estimators", 50, 500)    max_depth = trial.suggest_int("max_depth", 3, 30)    lr = trial.suggest_float("learning_rate", 1e-4, 1e-1, log=True)    model = GradientBoostingClassifier(        n_estimators=n_estimators, max_depth=max_depth, learning_rate=lr    )    score = cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc").mean()    return scorestudy = optuna.create_study(direction="maximize")study.optimize(objective, n_trials=100)print(study.best_params, study.best_value)

Search Strategies

Approaches to exploring hyperparameter space.

Grid Search- exhaustively tries every combination; guaranteed to find the best in the grid, but expensive
Random Search- samples combinations randomly; often finds good configs faster than grid search
Bayesian Optimization- builds a probabilistic model of the objective to choose promising next trials (e.g. Optuna, Hyperopt)
Successive Halving / Hyperband- allocates more resources to promising configs, prunes weak ones early
Population-Based Training- evolves a population of models and hyperparameters together during training
Manual/coarse-to-fine search- start with a wide range, narrow around good regions iteratively

Commonly Tuned Hyperparameters

Frequent tuning targets across model families.

learning_rate- step size for gradient-based updates; tune on a log scale
n_estimators- number of trees/boosting rounds in ensemble models
max_depth- maximum depth of a tree; controls model complexity
regularization (L1/L2, alpha)- penalizes large weights to reduce overfitting
batch_size- number of samples per gradient update in neural network training
dropout rate- fraction of units randomly dropped during training to prevent overfitting

Pro Tip

Use a log-uniform scale (not linear) when searching learning rate, regularization strength, or other parameters that span orders of magnitude — a linear grid oversamples large values and undersamples small ones.

Was this cheat sheet helpful?

Cross-Validation Cheat Sheet

Confusion Matrix & Metrics Cheat Sheet

Explore Topics

#HyperparameterTuning#HyperparameterTuningCheatSheet#DataScience#Intermediate#GridRandomSearch#BayesianOptimizationWithOptuna#SearchStrategies#CommonlyTunedHyperparameters#Algorithms#MachineLearning#CheatSheet#SkillVeris

Sri Hayavadhana Info-Tech

Professional Web Designing Services