Nonelinear Model

Decision Tree

• decision tree is a supervised prediction model
  it is usually a k-nary tree. each leaf denotes one classification or continuous value of the dependent variable.
• two types of dependent variable:
  • classification tree
  • regression tree
• how to build a decision tree:
  • two steps:
    feature prioritization
    feature splitting
  • Algorithnm Details - ID3 Algorithm
    at each iteration step, we split the feature which can give us the largest entropy gain 直观理解：减少偏见
    Definition of Entropy: Entropy of the entire information system:
    
    image.png
    
    
    image.png
• other tree algortithms:
  • C4.5 Algorithm: Gain Rate

  • CART: Gini impurity

    
    
    image.png