Optuna Hyperparameter Optimization Cheat Sheet
Define-by-run hyperparameter search with samplers, pruners, distributed studies, and integrations for PyTorch, LightGBM, and sklearn.
2 PagesIntermediateFeb 18, 2026
Define and Run a Study
Create an objective function and optimize it over a fixed number of trials.
python
import optunadef objective(trial): lr = trial.suggest_float("lr", 1e-5, 1e-1, log=True) n_layers = trial.suggest_int("n_layers", 1, 4) optimizer_name = trial.suggest_categorical("optimizer", ["Adam", "SGD"]) dropout = trial.suggest_float("dropout", 0.0, 0.5, step=0.1) model = build_model(n_layers, dropout) accuracy = train_and_eval(model, lr, optimizer_name) return accuracy # value optuna will maximize/minimizestudy = optuna.create_study(direction="maximize", study_name="mnist-cnn")study.optimize(objective, n_trials=100, timeout=600)print(study.best_params)print(study.best_value)
Samplers and Pruners
Configure the search strategy and enable early stopping of unpromising trials.
python
from optuna.samplers import TPESamplerfrom optuna.pruners import MedianPrunerstudy = optuna.create_study( direction="minimize", sampler=TPESampler(seed=42), pruner=MedianPruner(n_startup_trials=5, n_warmup_steps=10),)def objective(trial): for epoch in range(30): loss = train_one_epoch() trial.report(loss, step=epoch) if trial.should_prune(): raise optuna.TrialPruned() return loss
Persistent and Distributed Studies
Store trials in a relational database so multiple workers can optimize the same study in parallel.
bash
# create a persistent study backed by SQLite or PostgreSQLoptuna create-study --study-name "nlp-tuning" \ --storage "postgresql://user:pass@host/optuna_db"# each worker (on any machine) attaches to the same studypython train_worker.py # internally calls optuna.load_study(...)# inspect results from the CLIoptuna trials --study-name "nlp-tuning" --storage "postgresql://user:pass@host/optuna_db"
LightGBM Integration Example
Use suggested parameters directly inside a gradient boosting training loop.
python
import optunaimport lightgbm as lgbdef objective(trial): params = { "objective": "binary", "metric": "auc", "num_leaves": trial.suggest_int("num_leaves", 16, 256), "learning_rate": trial.suggest_float("learning_rate", 1e-3, 0.3, log=True), "feature_fraction": trial.suggest_float("feature_fraction", 0.5, 1.0), "bagging_fraction": trial.suggest_float("bagging_fraction", 0.5, 1.0), } gbm = lgb.train(params, train_set, valid_sets=[valid_set], callbacks=[lgb.early_stopping(50)]) preds = gbm.predict(X_valid) return roc_auc_score(y_valid, preds)study = optuna.create_study(direction="maximize")study.optimize(objective, n_trials=50)
Trial Suggestion API
The core methods used to define a search space inside an objective function.
- trial.suggest_float(name, low, high, log=False)- samples a continuous value, optionally on a log scale
- trial.suggest_int(name, low, high, step=1)- samples an integer, optionally stepped
- trial.suggest_categorical(name, choices)- samples from a fixed list of discrete options
- trial.report(value, step)- reports an intermediate value for pruning decisions
- trial.should_prune()- returns True if the current trial should be stopped early
- optuna.visualization.plot_optimization_history(study)- plots best-value progression across trials
Multi-Objective Optimization
Optimize competing objectives simultaneously (e.g. accuracy vs. latency) and inspect the resulting Pareto front.
python
import optunadef objective(trial): n_units = trial.suggest_int("n_units", 32, 512, log=True) model = build_model(n_units) accuracy = evaluate_accuracy(model) latency_ms = measure_inference_latency(model) return accuracy, latency_msstudy = optuna.create_study(directions=["maximize", "minimize"])study.optimize(objective, n_trials=200)# trials on the Pareto front — no other trial dominates them on both objectivesfor t in study.best_trials: print(t.values, t.params)
Warm-Starting with `enqueue_trial`
Seed a study with known-good configurations before the sampler starts exploring, so early trials aren't wasted.
python
study = optuna.create_study(direction="maximize")# force these exact param sets to run first, e.g. previous best configstudy.enqueue_trial({"lr": 3e-4, "n_layers": 3, "optimizer": "Adam", "dropout": 0.1})study.enqueue_trial({"lr": 1e-3, "n_layers": 2, "optimizer": "SGD", "dropout": 0.0})study.optimize(objective, n_trials=100) # enqueued trials run before sampler-chosen ones
PyTorch Lightning Pruning Integration
Report intermediate validation metrics each epoch so Optuna's pruner can kill a bad trial mid-training instead of waiting for it to finish.
python
import optunafrom optuna.integration import PyTorchLightningPruningCallbackimport pytorch_lightning as pldef objective(trial): model = LitModel(lr=trial.suggest_float("lr", 1e-5, 1e-1, log=True)) trainer = pl.Trainer( max_epochs=20, callbacks=[PyTorchLightningPruningCallback(trial, monitor="val_acc")], enable_checkpointing=False, ) trainer.fit(model) return trainer.callback_metrics["val_acc"].item()study = optuna.create_study(direction="maximize", pruner=optuna.pruners.HyperbandPruner())study.optimize(objective, n_trials=50)
Analyzing Results with `trials_dataframe`
Export completed trials to a pandas DataFrame for custom filtering, correlation analysis, or logging to an experiment tracker.
python
df = study.trials_dataframe(attrs=("number", "value", "params", "state", "duration"))completed = df[df["state"] == "COMPLETE"].sort_values("value", ascending=False)print(completed.head(10))# parameter importance (fANOVA-based) — which hyperparameters actually matterimportances = optuna.importance.get_param_importances(study)for name, score in importances.items(): print(f"{name}: {score:.3f}")
Sampler & Pruner Catalog
Beyond the TPE/Median defaults — pick based on search-space shape and trial budget.
- TPESampler- Tree-structured Parzen Estimator, Optuna's default; good general-purpose Bayesian sampler
- CmaEsSampler- covariance matrix adaptation, strong on continuous, non-separable search spaces
- GridSampler- exhaustive search over an explicit grid; deterministic, useful for small discrete spaces
- RandomSampler- pure random baseline; useful to sanity-check that TPE is actually beating chance
- HyperbandPruner- allocates more budget to promising trials via successive halving; needs trial.report() at multiple steps
- SuccessiveHalvingPruner- similar to Hyperband but with a single bracket; simpler, less overhead for smaller studies
- PatientPruner- wraps another pruner to require N consecutive bad reports before pruning, avoiding noisy early stops
Pro Tip
Use trial.suggest_float with log=True for learning rates and regularization strengths — sampling uniformly on a linear scale wastes most trials in a range that barely affects the result.
Was this cheat sheet helpful?
Explore Topics
#OptunaHyperparameterOptimization#OptunaHyperparameterOptimizationCheatSheet#DataScience#Intermediate#DefineAndRunAStudy#SamplersAndPruners#PersistentAndDistributedStudies#LightGBMIntegrationExample#MachineLearning#CheatSheet#SkillVeris
