CNN可视化

CNN可视化

1、查看感兴趣的任意层的方法
a、将图片放入网络
b、选择感兴趣的层，将该层中的感兴趣的神经元的上游梯度值设为1，其他神经元的值为0
c、反向传播至输入数据，求出dx

2、阴影图
a、将图片传入训练过的网络
b、前向传播至softmax层之前的那一层scores输出，确定该图片所属的类型，将该类值设置为1，其他的值设为0，作为上游梯度反向传播
c、由于是灰度图，而前向传播得到的dx可能有3channel，只需取3个channel中绝对值最大的那个
d、绘出灰度图

def compute_saliency_maps(X, y, model):  """  Compute a class saliency map using the model for images X and labels y.  Input:  - X: Input images, of shape (N, 3, H, W)  - y: Labels for X, of shape (N,)  - model: A PretrainedCNN that will be used to compute the saliency map.  Returns:  - saliency: An array of shape (N, H, W) giving the saliency maps for the input    images.  """  N, _, H, W = X.shape  saliency = np.zeros((N, H, W))  ##############################################################################  # TODO: Implement this function. You should use the forward and backward     #  # methods of the PretrainedCNN class, and compute gradients with respect to  #  # the unnormalized class score of the ground-truth classes in y.             #  ##############################################################################  out, cache = model.forward(X, start=None, end=None, mode = 'test')  dscores = np.zeros_like(out)  dscores[np.arange(N), y] = 1  dx, grads = model.backward(dscores, cache)  for n in range(N):    for h in range(H):        for w in range(W):            saliency[n, h, w] = np.max(np.abs((dx[n, :, h, w])))    #saliency[n,:,:] -= np.mean(saliency[n, :, :], axis=0, keepdims=True)  #saliency = np.abs(np.max(dx.reshape(-1,3), axis = 1)).reshape((N, H, W))  ##############################################################################  #                             END OF YOUR CODE                               #  ##############################################################################  return saliency

3、fool the net
给定任意的图片，我们可以通过一些设置让 网络误以为图片属于我们设定的那一类。和阴影图的绘制相似，我们只需最大化相应类的得分即可
a、将图片传入训练过的网络
b、前向传播至softmax层之前的那一层scores输出，将该我们希望图片的类值设置为1，其他的值设为0，作为上游梯度反向传播
c、梯度提升更新图片（设置正确的参数后应该可以在100次迭代内fool the net）

def make_fooling_image(X, target_y, model):  """  Generate a fooling image that is close to X, but that the model classifies  as target_y.  Inputs:  - X: Input image, of shape (1, 3, 64, 64)  - target_y: An integer in the range [0, 100)  - model: A PretrainedCNN  Returns:  - X_fooling: An image that is close to X, but that is classifed as target_y    by the model.  """  X_fooling = X.copy()  ##############################################################################  # TODO: Generate a fooling image X_fooling that the model will classify as   #  # the class target_y. Use gradient ascent on the target class score, using   #  # the model.forward method to compute scores and the model.backward method   #  # to compute image gradients.                                                #  #                                                                            #  # HINT: For most examples, you should be able to generate a fooling image    #  # in fewer than 100 iterations of gradient ascent.                           #  ##############################################################################  N, _, H, W = X.shape  for i in range(100):      scores = model.loss(X)[0]      print scores[target_y], np.max(scores)      if np.argmax(scores) == target_y:        break      out, cache = model.forward(X, start=None, end=None, mode = 'test')      dscores = np.zeros_like(out)      dscores[np.arange(N), target_y] = 1      out *= dscores      dx, grads = model.backward(out, cache)            X += 5000*dx  ##############################################################################  #                             END OF YOUR CODE                               #  ##############################################################################  return X

4、利用训练过的网络生成我们希望的类型图片

a、随机生成相应维度的原始数据图片（如64X64X3）
b、前向传播至softmax层之前的那一层scores输出，将该我们希望图片的类值设置为1，其他的值设为0，作为上游梯度反向传播
c、梯度提升更新图片（设置正确的参数后应该可以在100次迭代内fool the net）Undo the jitter、As a regularizer, clip the image、As a regularizer, periodically blur the image；

def create_class_visualization(target_y, model, **kwargs):  """  Perform optimization over the image to generate class visualizations.  Inputs:  - target_y: Integer in the range [0, 100) giving the target class  - model: A PretrainedCNN that will be used for generation  Keyword arguments:  - learning_rate: Floating point number giving the learning rate  - blur_every: An integer; how often to blur the image as a regularizer  - l2_reg: Floating point number giving L2 regularization strength on the image;    this is lambda in the equation above.  - max_jitter: How much random jitter to add to the image as regularization  - num_iterations: How many iterations to run for  - show_every: How often to show the image  """  learning_rate = kwargs.pop('learning_rate', 10000)  blur_every = kwargs.pop('blur_every', 1)  l2_reg = kwargs.pop('l2_reg', 1e-6)  max_jitter = kwargs.pop('max_jitter', 4)  num_iterations = kwargs.pop('num_iterations', 100)  show_every = kwargs.pop('show_every', 25)  X = np.random.randn(1, 3, 64, 64)  N = 1  for t in xrange(num_iterations):    # As a regularizer, add random jitter to the image    ox, oy = np.random.randint(-max_jitter, max_jitter+1, 2)    X = np.roll(np.roll(X, ox, -1), oy, -2)    dX = None    ############################################################################    # TODO: Compute the image gradient dX of the image with respect to the     #    # target_y class score. This should be similar to the fooling images. Also #    # add L2 regularization to dX and update the image X using the image       #    # gradient and the learning rate.                                          #    ############################################################################    out, cache = model.forward(X, start=None, end=None, mode = 'test')    dscores = np.zeros_like(out)    dscores[np.arange(N), target_y] = 1    dscores *= out    dx, grads = model.backward(dscores, cache)     X += learning_rate*(dx + 2*l2_reg*X)    ############################################################################    #                             END OF YOUR CODE                             #    ############################################################################    # Undo the jitter    X = np.roll(np.roll(X, -ox, -1), -oy, -2)    # As a regularizer, clip the image    X = np.clip(X, -data['mean_image'], 255.0 - data['mean_image'])    # As a regularizer, periodically blur the image    if t % blur_every == 0:      X = blur_image(X)    # Periodically show the image    if t % show_every == 0:      plt.imshow(deprocess_image(X, data['mean_image']))      plt.gcf().set_size_inches(3, 3)      plt.axis('off')      plt.show()  return X

5、feature inversion

主要的思路在于将loss损失部分的导数代入网络反向传播求出dx，不断更新X得到 表征特征的显示。

def invert_features(target_feats, layer, model, **kwargs):  """  Perform feature inversion in the style of Mahendran and Vedaldi 2015, using  L2 regularization and periodic blurring.  Inputs:  - target_feats: Image features of the target image, of shape (1, C, H, W);    we will try to generate an image that matches these features  - layer: The index of the layer from which the features were extracted  - model: A PretrainedCNN that was used to extract features  Keyword arguments:  - learning_rate: The learning rate to use for gradient descent  - num_iterations: The number of iterations to use for gradient descent  - l2_reg: The strength of L2 regularization to use; this is lambda in the    equation above.  - blur_every: How often to blur the image as implicit regularization; set    to 0 to disable blurring.  - show_every: How often to show the generated image; set to 0 to disable    showing intermediate reuslts.  Returns:  - X: Generated image of shape (1, 3, 64, 64) that matches the target features.  """  learning_rate = kwargs.pop('learning_rate', 10000)  num_iterations = kwargs.pop('num_iterations', 500)  l2_reg = kwargs.pop('l2_reg', 1e-7)  blur_every = kwargs.pop('blur_every', 1)  show_every = kwargs.pop('show_every', 50)  X = np.random.randn(1, 3, 64, 64)  #target_out, _ = model.forward(target_feats, start=0, end=layer, mode = 'test')  for t in xrange(num_iterations):    ############################################################################    # TODO: Compute the image gradient dX of the reconstruction loss with      #    # respect to the image. You should include L2 regularization penalizing    #    # large pixel values in the generated image using the l2_reg parameter;    #    # then update the generated image using the learning_rate from above.      #    ############################################################################        out, cache = model.forward(X, start=0, end=layer, mode = 'test')    dx, grads = model.backward(2*(out - target_feats), cache)     X -= learning_rate*(dx + 2*l2_reg*X)    ############################################################################    #                             END OF YOUR CODE                             #    ############################################################################    # As a regularizer, clip the image    X = np.clip(X, -data['mean_image'], 255.0 - data['mean_image'])    # As a regularizer, periodically blur the image    if (blur_every > 0) and t % blur_every == 0:      X = blur_image(X)    if (show_every > 0) and (t % show_every == 0 or t + 1 == num_iterations):      plt.imshow(deprocess_image(X, data['mean_image']))      plt.gcf().set_size_inches(3, 3)      plt.axis('off')      plt.title('t = %d' % t)      plt.show()

通过设置layer可以找出浅层和深层各个特征，但要注意的是层次越深，需要迭代次数越多
6、deep dream
创建一些有意思的图片

直观来说，只用将相关层的梯度直接设置为其激活单元的输出，反向传播更新网络

def deepdream(X, layer, model, **kwargs):  """  Generate a DeepDream image.  Inputs:  - X: Starting image, of shape (1, 3, H, W)  - layer: Index of layer at which to dream  - model: A PretrainedCNN object  Keyword arguments:  - learning_rate: How much to update the image at each iteration  - max_jitter: Maximum number of pixels for jitter regularization  - num_iterations: How many iterations to run for  - show_every: How often to show the generated image  """  X = X.copy()  learning_rate = kwargs.pop('learning_rate', 5.0)  max_jitter = kwargs.pop('max_jitter', 16)  num_iterations = kwargs.pop('num_iterations', 100)  show_every = kwargs.pop('show_every', 25)  for t in xrange(num_iterations):    # As a regularizer, add random jitter to the image    ox, oy = np.random.randint(-max_jitter, max_jitter+1, 2)    X = np.roll(np.roll(X, ox, -1), oy, -2)    dX = None    ############################################################################    # TODO: Compute the image gradient dX using the DeepDream method. You'll   #    # need to use the forward and backward methods of the model object to      #    # extract activations and set gradients for the chosen layer. After        #    # computing the image gradient dX, you should use the learning rate to     #    # update the image X.                                                      #    ############################################################################    out, cache = model.forward(X, start=0, end=layer, mode = 'test')    dx, grads = model.backward(out, cache)     X += learning_rate*dx    ############################################################################    #                             END OF YOUR CODE                             #    ############################################################################    # Undo the jitter    X = np.roll(np.roll(X, -ox, -1), -oy, -2)    # As a regularizer, clip the image    mean_pixel = data['mean_image'].mean(axis=(1, 2), keepdims=True)    X = np.clip(X, -mean_pixel, 255.0 - mean_pixel)    # Periodically show the image    if t == 0 or (t + 1) % show_every == 0:      img = deprocess_image(X, data['mean_image'], mean='pixel')      plt.imshow(img)      plt.title('t = %d' % (t + 1))      plt.gcf().set_size_inches(8, 8)      plt.axis('off')      plt.show()  return X

• 读取图片使用的函数

from scipy.misc import imreadimg = imread('kitten.jpg')#获得图片矩阵H, W, _ = img.shapeimg = imresize(img, (max_size, int(W * float(max_size) / H)))#变换图片的大小plt.imshow(img)#显示图片