机器学习：5.4 Stacking

@[toc]

  • Combine multiple base learners to reduce variance

    • Base learners can be different model types
    • Linearly combine base learners outputs by learned parameters

  • Widely used in competitions

  • bagging VS stacking

    • Bagging: bootstrap samples to get diversity

    • Stacking: different types of models extract different features

Multi-layer Stacking

  • Stacking base learners in multiple levels to reduce bias
    • Can use a different set of base learners at each level
  • Upper levels (e.g. L2) are trained on the outputs of the level below (e.g. L1)
    • Concatenating original inputs helps

Overfitting in Multi-layer Stacking

  • Train leaners from different levels on different data to alleviate
    overfitting

  • Split training data into A and B, train L_1 learners on A, run inference on B to generate training data for L_2 learners

  • Repeated k-fold bagging:

    • Train k models as in k-fold cross validation
    • Combine predictions of each model on out-of-fold data
    • Repeat step 1,2 by n times, average the n predictions of each example for the next level training
©著作权归作者所有,转载或内容合作请联系作者
平台声明：文章内容（如有图片或视频亦包括在内）由作者上传并发布，文章内容仅代表作者本人观点，简书系信息发布平台，仅提供信息存储服务。

推荐阅读更多精彩内容