Gradient Boosting
An ensemble technique that builds models sequentially, where each new model focuses on correcting the errors made by previous models. It combines many weak learners into a single strong learner.
Why It Matters
Gradient boosting (XGBoost, LightGBM, CatBoost) consistently wins ML competitions on tabular data and is the go-to algorithm for many production applications.
Example
XGBoost predicting customer churn by building 1000 small decision trees sequentially, each one focusing on the customers that previous trees got wrong.
Think of it like...
Like a relay team where each runner focuses on making up for the previous runner's weak spots — together they achieve a better overall time than any single runner.
Related Terms
XGBoost
Extreme Gradient Boosting — an optimized implementation of gradient boosting that is fast, accurate, and the most winning algorithm in machine learning competitions on tabular data.
LightGBM
Light Gradient Boosting Machine — Microsoft's gradient boosting framework optimized for speed and efficiency. LightGBM uses histogram-based splitting and leaf-wise growth for faster training.
Random Forest
An ensemble learning method that builds multiple decision trees during training and outputs the majority vote (classification) or average prediction (regression) of all the trees. The 'forest' of diverse trees is more robust than any single tree.
Decision Tree
A supervised learning algorithm that makes predictions by learning a series of if-then-else decision rules from the data. It creates a tree-like structure where each internal node tests a feature and each leaf provides a prediction.
Ensemble Learning
A strategy that combines multiple models to produce better predictions than any single model alone. Ensemble methods leverage the diversity of different models to reduce errors.