Hyperparameter Tuning Cheat Sheet
Strategies and code patterns for searching hyperparameter space, including grid search, random search, and Bayesian optimization with scikit-learn and Optuna.
2 PagesIntermediateMar 18, 2026
Grid & Random Search
Exhaustive and sampled hyperparameter search.
python
from sklearn.model_selection import GridSearchCV, RandomizedSearchCVfrom sklearn.ensemble import RandomForestClassifierfrom scipy.stats import randintparam_grid = { "n_estimators": [100, 200, 300], "max_depth": [None, 10, 20, 30], "min_samples_split": [2, 5, 10],}grid = GridSearchCV(RandomForestClassifier(), param_grid, cv=5, scoring="f1", n_jobs=-1)grid.fit(X_train, y_train)print(grid.best_params_, grid.best_score_)# Random search samples a fixed number of combinations -- scales betterparam_dist = { "n_estimators": randint(50, 500), "max_depth": randint(3, 50),}random_search = RandomizedSearchCV(RandomForestClassifier(), param_dist, n_iter=50, cv=5, scoring="f1", n_jobs=-1, random_state=42)random_search.fit(X_train, y_train)
Bayesian Optimization with Optuna
Sample promising trials using a probabilistic model.
python
import optunafrom sklearn.model_selection import cross_val_scorefrom sklearn.ensemble import GradientBoostingClassifierdef objective(trial): n_estimators = trial.suggest_int("n_estimators", 50, 500) max_depth = trial.suggest_int("max_depth", 3, 30) lr = trial.suggest_float("learning_rate", 1e-4, 1e-1, log=True) model = GradientBoostingClassifier( n_estimators=n_estimators, max_depth=max_depth, learning_rate=lr ) score = cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc").mean() return scorestudy = optuna.create_study(direction="maximize")study.optimize(objective, n_trials=100)print(study.best_params, study.best_value)
Search Strategies
Approaches to exploring hyperparameter space.
- Grid Search- exhaustively tries every combination; guaranteed to find the best in the grid, but expensive
- Random Search- samples combinations randomly; often finds good configs faster than grid search
- Bayesian Optimization- builds a probabilistic model of the objective to choose promising next trials (e.g. Optuna, Hyperopt)
- Successive Halving / Hyperband- allocates more resources to promising configs, prunes weak ones early
- Population-Based Training- evolves a population of models and hyperparameters together during training
- Manual/coarse-to-fine search- start with a wide range, narrow around good regions iteratively
Commonly Tuned Hyperparameters
Frequent tuning targets across model families.
- learning_rate- step size for gradient-based updates; tune on a log scale
- n_estimators- number of trees/boosting rounds in ensemble models
- max_depth- maximum depth of a tree; controls model complexity
- regularization (L1/L2, alpha)- penalizes large weights to reduce overfitting
- batch_size- number of samples per gradient update in neural network training
- dropout rate- fraction of units randomly dropped during training to prevent overfitting
Successive Halving Search
Prune weak candidates early and allocate more data/iterations to promising ones.
python
from sklearn.experimental import enable_halving_search_cv # noqafrom sklearn.model_selection import HalvingRandomSearchCVfrom sklearn.ensemble import RandomForestClassifierfrom scipy.stats import randintparam_dist = { "n_estimators": randint(50, 500), "max_depth": randint(3, 30), "min_samples_leaf": randint(1, 20),}search = HalvingRandomSearchCV( RandomForestClassifier(random_state=42), param_dist, resource="n_samples", factor=3, cv=5, scoring="f1", random_state=42,)search.fit(X_train, y_train)print(search.best_params_, search.best_score_)
Optuna Pruning of Unpromising Trials
Stop a trial early once its intermediate validation curve trails the best trials so far.
python
import optunafrom sklearn.ensemble import GradientBoostingClassifierfrom sklearn.metrics import roc_auc_scorefrom sklearn.model_selection import train_test_splitdef objective(trial): lr = trial.suggest_float("learning_rate", 1e-3, 0.3, log=True) n_estimators = trial.suggest_int("n_estimators", 50, 500) X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.2) model = GradientBoostingClassifier(learning_rate=lr, n_estimators=1, warm_start=True) for step in range(1, n_estimators + 1): model.n_estimators = step model.fit(X_tr, y_tr) auc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1]) trial.report(auc, step) if trial.should_prune(): raise optuna.TrialPruned() return aucstudy = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner())study.optimize(objective, n_trials=50)
Multi-Objective Optimization
Jointly optimize accuracy and inference latency to find the Pareto front instead of one scalar winner.
python
import optunaimport timefrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import cross_val_scoredef objective(trial): n_estimators = trial.suggest_int("n_estimators", 20, 400) max_depth = trial.suggest_int("max_depth", 2, 32) model = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth) accuracy = cross_val_score(model, X_train, y_train, cv=3, scoring="accuracy").mean() model.fit(X_train, y_train) start = time.perf_counter() model.predict(X_train[:1000]) latency = time.perf_counter() - start return accuracy, latencystudy = optuna.create_study(directions=["maximize", "minimize"])study.optimize(objective, n_trials=60)pareto_trials = study.best_trials # non-dominated (accuracy, latency) tradeoffs
Hyperparameter Importance Analysis
Quantify which hyperparameters actually drive score variance after a search completes.
python
import optuna# after study.optimize(...) has been runimportances = optuna.importance.get_param_importances(study)for param, importance in importances.items(): print(f"{param}: {importance:.3f}")# Visualize with optuna's built-in plots (requires plotly)fig = optuna.visualization.plot_param_importances(study)fig2 = optuna.visualization.plot_parallel_coordinate(study)
Optuna Samplers & Pruners
Building blocks for configuring how trials are proposed and cut short.
- TPESampler- tree-structured Parzen estimator, the default; models good vs. bad regions of the search space
- CmaEsSampler- covariance matrix adaptation, effective for continuous, correlated parameter spaces
- GridSampler- exhaustively evaluates a fixed grid, useful for reproducible small searches
- MedianPruner- prunes a trial if its intermediate value is worse than the median of prior trials at the same step
- HyperbandPruner- allocates a budget across brackets like successive halving, strong default for iterative models
- SuccessiveHalvingPruner- aggressive early stopping based on ASHA (asynchronous successive halving)
Pro Tip
Use a log-uniform scale (not linear) when searching learning rate, regularization strength, or other parameters that span orders of magnitude — a linear grid oversamples large values and undersamples small ones.
Was this cheat sheet helpful?
Explore Topics
#HyperparameterTuning#HyperparameterTuningCheatSheet#DataScience#Intermediate#GridRandomSearch#BayesianOptimizationWithOptuna#SearchStrategies#CommonlyTunedHyperparameters#Algorithms#MachineLearning#CheatSheet#SkillVeris
