@[toc]
Manual Hyperparameter Tuning
Start with a good baseline, e.g. default settings in high-quality toolkits, values reported in papers
Tune a value, retrain the model to see the changes
-
Repeat multiple times to gain insights about
Which hyperparameters are important
How sensitive the model to hyperparameters
What are the good ranges
Needs careful experiment management
-
Save your training logs and hyperparameters to compare, share and
reproduce laterThe simplest way is saving logs in text and put key metrics in Excel
Better options exist, e.g. tenesorboard and weights & bias
-
Reproducing is hard, it relates to
Environment (hardware & library)
Code
Randomness (seed)
Automated Machine Learning (AutoML)
- Automate every step in applying ML to solve real-world problems: data cleaning, feature extraction, model selection…
- Hyperparameter optimization (HPO):find a good set of hyperparameters
through search algorithms - Neural architecture search (NAS):construct a good neural network model
Summary
- Hyperparameter tuning aims to find a set of good values
- It’s time consuming as data preprocessing
- There is a trend to use algorithm for tuning
