Model pruning knowledge points summary

2024-07-12

Model pruning knowledge points summary

Pruning isDeep Learning ModelsTwo common optimization techniques to reduce model complexity and improve inference speed, suitable for resource-constrained environments.

Pruning

Pruning is a method to reduce the model size and computational effort by removing unimportant or redundant parameters in the model. Pruning is usually divided into the following types:

1. Weight Pruning

Weight pruning is to reduce the number of model parameters by removing elements close to zero in the weight matrix. Common methods are:

Unstructured Pruning: Remove small weights from the weight matrix one by one.
Structured Pruning: Remove weights by a specific structure, such as an entire row or column.

Example:

import torch

# 假设有一个全连接层
fc = torch.nn.Linear(100, 100)

# 获取权重矩阵
weights = fc.weight.data.abs()

# 设定剪枝阈值
threshold = 0.01

# 应用剪枝
mask = weights > threshold
fc.weight.data *= mask
1
2
3
4
5
6
7
8
9
10
11
12
13
14

2. Channel Pruning

Channel pruning is mainly used forConvolutional Neural Networks, by removing unimportant channels in the convolutional layer to reduce the amount of computation. Common methods include:

Based on importance score: Calculate the importance score of each channel and remove channels with low scores.
Based on sparsity: By adding sparse regularization terms, some channels are naturally made sparse during the training process and then pruned.

import torch
import torch.nn as nn

class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(64, 128, kernel_size=3, padding=1)
    
    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        return x

model = ConvNet()

# 获取卷积层的权重
weights = model.conv1.weight.data.abs()

# 计算每个通道的L1范数
channel_importance = torch.sum(weights, dim=[1, 2, 3])

# 设定剪枝阈值
threshold = torch.topk(channel_importance, k=32, largest=True).values[-1]

# 应用剪枝
mask = channel_importance > threshold
model.conv1.weight.data *= mask.view(-1, 1, 1, 1)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

3. Layer Pruning

Layer pruning is to remove entire network layers to reduce the computational depth of the model. This method is more aggressive and is usually used in conjunction with model architecture search (NAS).

import torch.nn as nn

class LayerPrunedNet(nn.Module):
    def __init__(self, use_layer=True):
        super(LayerPrunedNet, self).__init__()
        self.use_layer = use_layer
        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(64, 128, kernel_size=3, padding=1)
    
    def forward(self, x):
        x = self.conv1(x)
        if self.use_layer:
            x = self.conv2(x)
        return x

# 初始化网络，选择是否使用第二层
model = LayerPrunedNet(use_layer=False)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

Technology Sharing