@[toc]
-
Combine multiple base learners to reduce variance
- Base learners can be different model types
- Linearly combine base learners outputs by learned parameters
Widely used in competitions
-
bagging VS stacking
Bagging: bootstrap samples to get diversity
Stacking: different types of models extract different features
Multi-layer Stacking
- Stacking base learners in multiple levels to reduce bias
- Can use a different set of base learners at each level
- Upper levels (e.g. L2) are trained on the outputs of the level below (e.g. L1)
- Concatenating original inputs helps
Overfitting in Multi-layer Stacking
Train leaners from different levels on different data to alleviate
overfittingSplit training data into A and B, train
learners on A, run inference on B to generate training data for
learners
-
Repeated k-fold bagging:
- Train k models as in k-fold cross validation
- Combine predictions of each model on out-of-fold data
- Repeat step 1,2 by n times, average the n predictions of each example for the next level training
