CNN卷积神经网络内存占用计算

转自：http://cs231n.github.io/convolutional-networks/#case

http://n2value.com/blog/memory-requirements-for-convolutional-neural-network-analysis-of-brain-mri/

VGGNet in detail. Lets break down the VGGNet in more detail as a case study. The whole VGGNet is composed of CONV layers that perform 3x3 convolutions with stride 1 and pad 1, and of POOL layers that perform 2x2 max pooling with stride 2 (and no padding). We can write out the size of the representation at each step of the processing and keep track of both the representation size and the total number of weights:

INPUT: [224x224x3]        memory:  224*224*3=150K   weights: 0CONV3-64: [224x224x64]  memory:  224*224*64=3.2M   weights: (3*3*3)*64 = 1,728CONV3-64: [224x224x64]  memory:  224*224*64=3.2M   weights: (3*3*64)*64 = 36,864POOL2: [112x112x64]  memory:  112*112*64=800K   weights: 0CONV3-128: [112x112x128]  memory:  112*112*128=1.6M   weights: (3*3*64)*128 = 73,728CONV3-128: [112x112x128]  memory:  112*112*128=1.6M   weights: (3*3*128)*128 = 147,456POOL2: [56x56x128]  memory:  56*56*128=400K   weights: 0CONV3-256: [56x56x256]  memory:  56*56*256=800K   weights: (3*3*128)*256 = 294,912CONV3-256: [56x56x256]  memory:  56*56*256=800K   weights: (3*3*256)*256 = 589,824CONV3-256: [56x56x256]  memory:  56*56*256=800K   weights: (3*3*256)*256 = 589,824POOL2: [28x28x256]  memory:  28*28*256=200K   weights: 0CONV3-512: [28x28x512]  memory:  28*28*512=400K   weights: (3*3*256)*512 = 1,179,648CONV3-512: [28x28x512]  memory:  28*28*512=400K   weights: (3*3*512)*512 = 2,359,296CONV3-512: [28x28x512]  memory:  28*28*512=400K   weights: (3*3*512)*512 = 2,359,296POOL2: [14x14x512]  memory:  14*14*512=100K   weights: 0CONV3-512: [14x14x512]  memory:  14*14*512=100K   weights: (3*3*512)*512 = 2,359,296CONV3-512: [14x14x512]  memory:  14*14*512=100K   weights: (3*3*512)*512 = 2,359,296CONV3-512: [14x14x512]  memory:  14*14*512=100K   weights: (3*3*512)*512 = 2,359,296POOL2: [7x7x512]  memory:  7*7*512=25K  weights: 0FC: [1x1x4096]  memory:  4096  weights: 7*7*512*4096 = 102,760,448FC: [1x1x4096]  memory:  4096  weights: 4096*4096 = 16,777,216FC: [1x1x1000]  memory:  1000 weights: 4096*1000 = 4,096,000TOTAL memory: 24M * 4 bytes ~= 93MB / image (only forward! ~*2 for bwd)TOTAL params: 138M parameters

As is common with Convolutional Networks, notice that most of the memory (and also compute time) is used in the early CONV layers, and that most of the parameters are in the last FC layers. In this particular case, the first FC layer contains 100M weights, out of a total of 140M.