XGBoost Cheat Sheet
XGBoost cheat sheet covering the scikit-learn and native training APIs, key hyperparameters, early stopping, and feature importance.
2 PagesIntermediateMar 22, 2026
Scikit-learn API
Familiar fit/predict interface.
python
from xgboost import XGBClassifiermodel = XGBClassifier( n_estimators=300, max_depth=6, learning_rate=0.05, subsample=0.8, colsample_bytree=0.8, eval_metric="logloss", random_state=42,)model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)preds = model.predict(X_test)proba = model.predict_proba(X_test)
Native DMatrix API
Lower-level API with more control.
python
import xgboost as xgbdtrain = xgb.DMatrix(X_train, label=y_train)dval = xgb.DMatrix(X_val, label=y_val)params = {"objective": "binary:logistic", "max_depth": 6, "eta": 0.05}bst = xgb.train( params, dtrain, num_boost_round=500, evals=[(dval, "validation")], early_stopping_rounds=20,)bst.save_model("model.json")
Key Hyperparameters
Parameters that matter most for tuning.
- n_estimators- number of boosting rounds (trees)
- max_depth- max tree depth, controls overfitting
- learning_rate (eta)- shrinks each tree's contribution
- subsample- fraction of rows sampled per tree
- colsample_bytree- fraction of columns sampled per tree
- reg_alpha / reg_lambda- L1/L2 regularization on leaf weights
- early_stopping_rounds- stop when the validation metric stops improving
Feature Importance
Inspect and plot which features matter.
python
import matplotlib.pyplot as pltfrom xgboost import plot_importanceplot_importance(model, max_num_features=15, importance_type="gain")plt.show()importances = model.feature_importances_
Custom Objective & Eval Functions
Define your own gradient/hessian for specialized loss functions.
python
import numpy as npdef weighted_logloss_obj(preds, dtrain): labels = dtrain.get_label() p = 1.0 / (1.0 + np.exp(-preds)) weight = np.where(labels == 1, 3.0, 1.0) # upweight positives grad = weight * (p - labels) hess = weight * p * (1.0 - p) return grad, hessdef custom_f1_eval(preds, dtrain): labels = dtrain.get_label() p = 1.0 / (1.0 + np.exp(-preds)) pred_labels = (p > 0.5).astype(int) tp = ((pred_labels == 1) & (labels == 1)).sum() fp = ((pred_labels == 1) & (labels == 0)).sum() fn = ((pred_labels == 0) & (labels == 1)).sum() f1 = 2 * tp / (2 * tp + fp + fn + 1e-9) return "f1", f1bst = xgb.train(params, dtrain, num_boost_round=300, obj=weighted_logloss_obj, custom_metric=custom_f1_eval, evals=[(dval, "val")], maximize=True)
Monotonic & Interaction Constraints
Enforce domain knowledge on how features may relate to the target.
python
params = { "objective": "reg:squarederror", # +1 = must increase, -1 = must decrease, 0 = unconstrained "monotone_constraints": "(1,-1,0,0)", # only allow interactions within these feature-index groups "interaction_constraints": "[[0,1],[2,3]]", "max_depth": 6,}bst = xgb.train(params, dtrain, num_boost_round=400)# Sklearn API equivalentfrom xgboost import XGBRegressormodel = XGBRegressor(monotone_constraints=(1, -1, 0, 0))
SHAP Values for Model Explainability
Decompose predictions into per-feature contributions.
python
import shapexplainer = shap.TreeExplainer(model)shap_values = explainer.shap_values(X_test)shap.summary_plot(shap_values, X_test) # global feature impactshap.force_plot(explainer.expected_value, shap_values[0], X_test.iloc[0]) # single prediction# Native pred_contribs (no external dependency)contribs = bst.predict(dtrain, pred_contribs=True) # last col = bias term
DART Booster, GPU Training & xgb.cv
Dropout trees, GPU acceleration, and built-in cross-validation.
python
# DART: drops trees during boosting to reduce overfittingdart_params = { "booster": "dart", "objective": "binary:logistic", "rate_drop": 0.1, "skip_drop": 0.5,}# GPU-accelerated histogram traininggpu_params = {"tree_method": "hist", "device": "cuda"}# Built-in k-fold CV with early stoppingcv_results = xgb.cv( params, dtrain, num_boost_round=1000, nfold=5, metrics="auc", early_stopping_rounds=25, seed=42,)best_rounds = cv_results["test-auc-mean"].idxmax() + 1
Advanced API Reference
Lesser-known knobs for production and specialized modeling.
- tree_method='hist'- histogram-based split finding, the default fast method for large data
- missing=np.nan- DMatrix constructor arg controlling how sparse/missing values route through splits
- num_parallel_tree- builds a random-forest ensemble at each boosting round
- grow_policy='lossguide'- leaf-wise growth similar to LightGBM instead of depth-wise
- callbacks=[xgb.callback.EarlyStopping(...)]- composable training callbacks (checkpointing, LR schedules)
- bst.save_model('model.ubj')- universal binary JSON format, preferred over pickling for portability
- predict(..., iteration_range=(0, n))- predict using only the first n boosted trees
Pro Tip
Set early_stopping_rounds together with an eval_set so training halts automatically once the validation metric plateaus — this both prevents overfitting and saves training time.
Was this cheat sheet helpful?
Explore Topics
#XGBoost#XGBoostCheatSheet#DataScience#Intermediate#ScikitLearnAPI#NativeDMatrixAPI#KeyHyperparameters#FeatureImportance#MachineLearning#APIs#CheatSheet#SkillVeris
