AutoML Basics Cheat Sheet
Automate model selection, hyperparameter tuning, and pipeline search with AutoML libraries like Auto-sklearn, FLAML, and AutoGluon.
FLAML Quickstart
Fit an AutoML model that searches algorithms and hyperparameters within a time budget.
from flaml import AutoMLautoml = AutoML()automl.fit( X_train, y_train, task="classification", time_budget=120, # seconds metric="roc_auc",)print("Best estimator:", automl.best_estimator)print("Best config:", automl.best_config)preds = automl.predict(X_test)
AutoGluon Tabular Predictor
Train and ensemble multiple model families on a tabular dataset with one call.
from autogluon.tabular import TabularPredictorpredictor = TabularPredictor(label="target", eval_metric="f1").fit( train_data=train_df, time_limit=600, presets="best_quality",)leaderboard = predictor.leaderboard(test_df, silent=True)predictions = predictor.predict(test_df)
Auto-sklearn Classifier
Run a Bayesian-optimization-driven search over sklearn pipelines with meta-learning warm starts.
import autosklearn.classificationclf = autosklearn.classification.AutoSklearnClassifier( time_left_for_this_task=300, per_run_time_limit=30, ensemble_size=10,)clf.fit(X_train, y_train)print(clf.leaderboard())y_pred = clf.predict(X_test)
When to Use Which Tool
Quick guidance for picking an AutoML library based on constraints.
- FLAML- fastest, lightweight, good default for tight time budgets
- AutoGluon- best raw accuracy via stacked ensembles, heavier compute cost
- Auto-sklearn- strong meta-learning warm starts, sklearn-native pipelines
- H2O AutoML- scales well on Spark/large datasets, good leaderboard tooling
- time_budget / time_left_for_this_task- wall-clock cap on the whole search, not per model
Custom Search Space with Optuna Pruning
Define your own hyperparameter search space and use median pruning to kill unpromising trials early.
import optunafrom sklearn.ensemble import GradientBoostingClassifierfrom sklearn.model_selection import cross_val_scoredef objective(trial): params = { "n_estimators": trial.suggest_int("n_estimators", 50, 500), "max_depth": trial.suggest_int("max_depth", 2, 10), "learning_rate": trial.suggest_float("learning_rate", 1e-3, 0.3, log=True), "subsample": trial.suggest_float("subsample", 0.5, 1.0), } model = GradientBoostingClassifier(**params) score = cross_val_score(model, X_train, y_train, cv=3, scoring="roc_auc").mean() return scorestudy = optuna.create_study( direction="maximize", pruner=optuna.pruners.MedianPruner(n_warmup_steps=5),)study.optimize(objective, n_trials=200, timeout=600)print(study.best_params, study.best_value)
Automated Feature Engineering with Featuretools
Run Deep Feature Synthesis to generate engineered features from relational tables before AutoML search.
import featuretools as ftes = ft.EntitySet(id="orders")es = es.add_dataframe(dataframe_name="orders", dataframe=orders_df, index="order_id", time_index="order_date")es = es.add_dataframe(dataframe_name="customers", dataframe=customers_df, index="customer_id")es = es.add_relationship("customers", "customer_id", "orders", "customer_id")feature_matrix, feature_defs = ft.dfs( entityset=es, target_dataframe_name="customers", agg_primitives=["mean", "sum", "count", "trend"], trans_primitives=["day", "month", "weekday"], max_depth=2,)print(feature_matrix.shape) # engineered feature table ready for AutoML.fit()
Customize AutoGluon's Stacking/Bagging
Control the multi-layer stack ensemble depth and bagging folds instead of relying on the default preset.
from autogluon.tabular import TabularPredictorpredictor = TabularPredictor(label="target", eval_metric="roc_auc").fit( train_data=train_df, num_bag_folds=8, # k-fold bagging per base model num_bag_sets=1, num_stack_levels=2, # depth of the stacked ensemble hyperparameters={ "GBM": {}, "CAT": {}, "XGB": {}, "NN_TORCH": {"num_epochs": 20}, }, time_limit=1200,)print(predictor.fit_summary())print(predictor.leaderboard(extra_info=True).head())
Explain the AutoML Winner with SHAP
Attach model-agnostic SHAP explanations to the best pipeline AutoML selected, since search itself gives no interpretability.
import shapbest_model = predictor.get_model_best()predict_fn = lambda x: predictor.predict_proba(x, model=best_model).valuesexplainer = shap.KernelExplainer(predict_fn, X_train.sample(100))shap_values = explainer.shap_values(X_test.sample(50))shap.summary_plot(shap_values, X_test.sample(50))# use this to sanity-check that the top AutoML model isn't relying on a leaky feature
What's Actually Happening Inside AutoML Search
The optimization techniques AutoML libraries combine under the hood, beyond just 'try lots of models'.
- Bayesian optimization- builds a surrogate model of the score function to pick the next hyperparameters to try, smarter than grid/random search
- Successive halving / ASHA- allocates a small budget to many configs, then doubles the budget only for the best-performing subset
- Meta-learning warm start- initializes the search near configs that worked well on similar past datasets (used by Auto-sklearn, AutoGluon)
- CASH problem- Combined Algorithm Selection and Hyperparameter optimization, the formal search problem AutoML solves
- Stacked generalization- trains a meta-model on the out-of-fold predictions of base models, usually the biggest accuracy lever
- Pipeline search- searches over preprocessing steps (imputation, scaling, encoding) jointly with the model, not just model hyperparameters
Treat AutoML output as a strong baseline, not a final model — always inspect the leaderboard's top 3 candidates manually, since the single 'best' model by validation score is sometimes the most overfit one.
