image

Introduction

• learning without forgetting与transfer learning, multi-task learning相似。
• Currently, there are three common approaches (Figs. 1, 2(b-d)) to learning θn while benefiting from previously learned θs, and they differ mostly on which parameters are unchanged
• Feature Extraction: θs and θo are unchanged
• Fine-tuning: θs and θn are optimized for the new task, while θo is fixed. A low learning rate is typically used to prevent large drift in θs.
• Fine-tuning FC: only top fully connected layers are fine-tuned
• Joint Training：All parameters θs, θo, θn are jointly optimized. This method’s performance may be seen as an upper bound of what our proposed method can achieve.
  Learning without Forgetting的优点：

1. 分类性能优于feature extraction，fine-tuning，其他最近的方法。
2. 计算效率高，训练时间比joint training和fine-tuning快
3. 简单的部署，一旦学习了一个任务，就不需要保留或重新应用训练数据来保持适应网络的性能。

Related Work

• Multi-task learning
• Domain Adaptation：the same tasks but in different domains
• transfer learning：different tasks
• Learning without Forgetting approach could be seen as a combination of Distillation Networks and fine-tuning.
• Feature Extraction: 不修改原来的network，允许新任务从以前任务中学习的复杂特性中获益。但是，这些特性并不专门用于新任务，通常可以通过微调进行改进
• Fine-tuning: 经常使用小的学习率，有时部分network被固定防止overfitting。
• Multitask learning：通常，网络的底层是共享的，而顶层是特定于任务的。多任务学习需要所有任务的数据