Approach

Given a trained CNN model, we propose an iterative two-step algorithm to effectively prune each layer, by a LASSO regression based channel selection and least square reconstruction. We alternatively take two steps. Further, we approximate the network layer-by-layer, with accumulated error accounted.

Formally, to prune a feature map with c channels, we consider applying n×c×k_h×k_w convolutional filters W on N×c×k_h×k_w input volumes X sampled from this feature map, which produces N×n output matrix Y. Here, N is the number of samples, n is the number of output channels.

To prune the input channels from c to desired c0 , while minimizing reconstruction error, we formulate our problem as follow:

Now we solve this problem in two folds. First, we fix W, solve for channel selection. Second, we fix Β, solve W to reconstruct error.

We alternatively optimize (i) and (ii). In the beginning, W is initialized from the trained model, λ= 0, namely no penalty, and ||B|| = c. We gradually increase λ. For each change of λ, we iterate these two steps until ||B|| is stable.

Experiment

References:
Channel Pruning for Accelerating Very Deep Neural Networks, Yihui He, 2017, ICCV