Hyperparameter Tuning
The process of systematically searching for the best combination of hyperparameters for a model. Since hyperparameters are set before training, finding optimal values requires experimentation.
Why It Matters
Proper hyperparameter tuning can boost model performance by 10-30% with the same data and architecture. Automated tuning saves significant data science time.
Example
Using grid search to try all combinations of learning rates [0.001, 0.01, 0.1] and batch sizes [16, 32, 64], evaluating each combination on validation data.
Think of it like...
Like a chef experimenting with oven temperatures and cooking times to find the perfect combination for a recipe — systematic testing beats guessing.
Related Terms
Hyperparameter
Settings that are configured before training begins and control how the model learns, as opposed to parameters which are learned during training. Examples include learning rate, batch size, and number of layers.
Bayesian Optimization
A sequential optimization strategy for finding the best hyperparameters by building a probabilistic model of the objective function and using it to select the most promising configurations to evaluate.
Cross-Validation
A model evaluation technique that splits data into multiple folds, trains on some folds and tests on the held-out fold, repeating so every fold serves as the test set. It provides a robust estimate of model performance.