循环神经网络
来源:互联网 发布:孙叔敖之知翻译 编辑:程序博客网 时间:2024/06/05 11:45
1、RNN
RNN的结构是十分灵活的,如下图所示:
其中从左到右的结构分别可以用来作为神经网络、图形标注、情感分析、机器翻译、视频分类。
a、Vanilla RNN
b、LSTM
原始的RNN有很大的缺点,在训练序列较长的模型时,容易造成梯度消失和梯度爆炸的情况发生(不断进行矩阵的相乘)。因此,通常实际中使用RNN的变种LSTM(Long Short Term Memory),即将原始RNN的更新策略更改为一种叫做门机制。
其具体的思路是:与RNN类似,在每一步迭代中,需要接受一个输入Xt(D维),同时还有上一个隐含单元的数据Ht-1(H维),除此之外,LSTM还保留了一个Cell Stat Ct-1(H维)。LSTM的参数包括输入到隐含层的矩阵Wx(4HxD)
,隐含层到隐含层的矩阵Wh(4HxH),以及偏置向量b(4H)
在每一步的时限内,与原始RNN类似,先通过公式
然后按照公式进行更新
总结:
附cs231n代码
rnn_layer.py
import numpy as np"""This file defines layer types that are commonly used for recurrent neuralnetworks."""def tanh(x): return 2*sigmoid(2*x) - 1def rnn_step_forward(x, prev_h, Wx, Wh, b): """ Run the forward pass for a single timestep of a vanilla RNN that uses a tanh activation function. The input data has dimension D, the hidden state has dimension H, and we use a minibatch size of N. Inputs: - x: Input data for this timestep, of shape (N, D). - prev_h: Hidden state from previous timestep, of shape (N, H) - Wx: Weight matrix for input-to-hidden connections, of shape (D, H) - Wh: Weight matrix for hidden-to-hidden connections, of shape (H, H) - b: Biases of shape (H,) Returns a tuple of: - next_h: Next hidden state, of shape (N, H) - cache: Tuple of values needed for the backward pass. """ next_h, cache = None, None ############################################################################## # TODO: Implement a single forward step for the vanilla RNN. Store the next # # hidden state and any values you need for the backward pass in the next_h # # and cache variables respectively. # ############################################################################## out = np.dot(x, Wx) + np.dot(prev_h, Wh) + b next_h = tanh(out) cache = x, prev_h, Wx, Wh, b, next_h ############################################################################## # END OF YOUR CODE # ############################################################################## return next_h, cachedef rnn_step_backward(dnext_h, cache): """ Backward pass for a single timestep of a vanilla RNN. Inputs: - dnext_h: Gradient of loss with respect to next hidden state (N, H) - cache: Cache object from the forward pass Returns a tuple of: - dx: Gradients of input data, of shape (N, D) - dprev_h: Gradients of previous hidden state, of shape (N, H) - dWx: Gradients of input-to-hidden weights, of shape (N, H) - dWh: Gradients of hidden-to-hidden weights, of shape (H, H) - db: Gradients of bias vector, of shape (H,) """ dx, dprev_h, dWx, dWh, db = None, None, None, None, None ############################################################################## # TODO: Implement the backward pass for a single step of a vanilla RNN. # # # # HINT: For the tanh function, you can compute the local derivative in terms # # of the output value from tanh. # ############################################################################## N, H = dnext_h.shape x, pre_h, Wx, Wh, b, out = cache d_temp = dnext_h*(1 - out**2) #NXH dx = np.dot(d_temp, Wx.T) dprev_h = np.dot(d_temp, Wh.T) dWx = np.dot(x.T, d_temp) dWh = np.dot(pre_h.T, d_temp) db = np.sum(d_temp, axis=0) ############################################################################## # END OF YOUR CODE # ############################################################################## return dx, dprev_h, dWx, dWh, dbdef rnn_forward(x, h0, Wx, Wh, b): """ Run a vanilla RNN forward on an entire sequence of data. We assume an input sequence composed of T vectors, each of dimension D. The RNN uses a hidden size of H, and we work over a minibatch containing N sequences. After running the RNN forward, we return the hidden states for all timesteps. Inputs: - x: Input data for the entire timeseries, of shape (N, T, D). - h0: Initial hidden state, of shape (N, H) - Wx: Weight matrix for input-to-hidden connections, of shape (D, H) - Wh: Weight matrix for hidden-to-hidden connections, of shape (H, H) - b: Biases of shape (H,) Returns a tuple of: - h: Hidden states for the entire timeseries, of shape (N, T, H). - cache: Values needed in the backward pass """ h, cache = None, None ############################################################################## # TODO: Implement forward pass for a vanilla RNN running on a sequence of # # input data. You should use the rnn_step_forward function that you defined # # above. # ############################################################################## N, T, D = x.shape N, H = h0.shape h = np.zeros((N, T, H)) cache = [] prev_h = h0 for t in range(T): prev_h, cac = rnn_step_forward(x[:,t,:], prev_h, Wx, Wh, b) h[:,t,:] = prev_h.copy() cache.append(cac) ############################################################################## # END OF YOUR CODE # ############################################################################## return h, cachedef rnn_backward(dh, cache): """ Compute the backward pass for a vanilla RNN over an entire sequence of data. Inputs: - dh: Upstream gradients of all hidden states, of shape (N, T, H) Returns a tuple of: - dx: Gradient of inputs, of shape (N, T, D) - dh0: Gradient of initial hidden state, of shape (N, H) - dWx: Gradient of input-to-hidden weights, of shape (D, H) - dWh: Gradient of hidden-to-hidden weights, of shape (H, H) - db: Gradient of biases, of shape (H,) """ dx, dh0, dWx, dWh, db = None, None, None, None, None ############################################################################## # TODO: Implement the backward pass for a vanilla RNN running an entire # # sequence of data. You should use the rnn_step_backward function that you # # defined above. # ############################################################################## N, T, H = dh.shape D, _ = cache[0][2].shape dprev_ht = np.zeros((N, H)) dx = np.zeros((N, T, D)) dh0 = np.zeros((N, H)) dWx = np.zeros((D, H)) dWh = np.zeros((H, H)) db = np.zeros(H) for t in reversed(range(T)): dx_t, dprev_ht, dWx_t, dWh_t, db_t = rnn_step_backward(dprev_ht+dh[:,t,:]\ , cache[t]) dx[:,t,:] = dx_t.copy() dWx += dWx_t dWh += dWh_t db += db_t dh0 = dprev_ht ############################################################################## # END OF YOUR CODE # ############################################################################## return dx, dh0, dWx, dWh, dbdef word_embedding_forward(x, W): """ Forward pass for word embeddings. We operate on minibatches of size N where each sequence has length T. We assume a vocabulary of V words, assigning each to a vector of dimension D. Inputs: - x: Integer array of shape (N, T) giving indices of words. Each element idx of x muxt be in the range 0 <= idx < V. - W: Weight matrix of shape (V, D) giving word vectors for all words. Returns a tuple of: - out: Array of shape (N, T, D) giving word vectors for all input words. - cache: Values needed for the backward pass """ out, cache = None, None ############################################################################## # TODO: Implement the forward pass for word embeddings. # # # # HINT: This should be very simple. # ############################################################################## N, T = x.shape V, D = W.shape out = np.zeros((N, T, D)) for i in range(N): for t in range(T): out[i, t, :] = W[x[i, t]] cache = x, W ############################################################################## # END OF YOUR CODE # ############################################################################## return out, cachedef word_embedding_backward(dout, cache): """ Backward pass for word embeddings. We cannot back-propagate into the words since they are integers, so we only return gradient for the word embedding matrix. HINT: Look up the function np.add.at Inputs: - dout: Upstream gradients of shape (N, T, D) - cache: Values from the forward pass Returns: - dW: Gradient of word embedding matrix, of shape (V, D). """ dW = None ############################################################################## # TODO: Implement the backward pass for word embeddings. # # # # HINT: Look up the function np.add.at # ############################################################################## N, T, D = dout.shape x, W = cache V, D = W.shape dW = np.zeros(W.shape) np.add.at(dW, x.reshape(1,-1), dout.reshape(-1, D)) ############################################################################## # END OF YOUR CODE # ############################################################################## return dWdef sigmoid(x): """ A numerically stable version of the logistic sigmoid function. """ pos_mask = (x >= 0) neg_mask = (x < 0) z = np.zeros_like(x) z[pos_mask] = np.exp(-x[pos_mask]) z[neg_mask] = np.exp(x[neg_mask]) top = np.ones_like(x) top[neg_mask] = z[neg_mask] return top / (1 + z)def lstm_step_forward(x, prev_h, prev_c, Wx, Wh, b): """ Forward pass for a single timestep of an LSTM. The input data has dimension D, the hidden state has dimension H, and we use a minibatch size of N. Inputs: - x: Input data, of shape (N, D) - prev_h: Previous hidden state, of shape (N, H) - prev_c: previous cell state, of shape (N, H) - Wx: Input-to-hidden weights, of shape (D, 4H) - Wh: Hidden-to-hidden weights, of shape (H, 4H) - b: Biases, of shape (4H,) Returns a tuple of: - next_h: Next hidden state, of shape (N, H) - next_c: Next cell state, of shape (N, H) - cache: Tuple of values needed for backward pass. """ next_h, next_c, cache = None, None, None ############################################################################# # TODO: Implement the forward pass for a single timestep of an LSTM. # # You may want to use the numerically stable sigmoid implementation above. # ############################################################################# N, D = x.shape N, H = prev_h.shape a = np.dot(x, Wx) + np.dot(prev_h, Wh) + b #NX4H i = sigmoid(a[:, :H]) f = sigmoid(a[:, H:2*H]) o = sigmoid(a[:, 2*H:3*H]) g = tanh(a[:, 3*H:]) #NXH next_c = prev_c*f + i*g next_h = o*tanh(next_c) cache = x,f, i, g, o, next_c, next_h, prev_h, Wx, Wh,prev_c ############################################################################## # END OF YOUR CODE # ############################################################################## return next_h, next_c, cachedef lstm_step_backward(dnext_h, dnext_c, cache): """ Backward pass for a single timestep of an LSTM. Inputs: - dnext_h: Gradients of next hidden state, of shape (N, H) - dnext_c: Gradients of next cell state, of shape (N, H) - cache: Values from the forward pass Returns a tuple of: - dx: Gradient of input data, of shape (N, D) - dprev_h: Gradient of previous hidden state, of shape (N, H) - dprev_c: Gradient of previous cell state, of shape (N, H) - dWx: Gradient of input-to-hidden weights, of shape (D, 4H) - dWh: Gradient of hidden-to-hidden weights, of shape (H, 4H) - db: Gradient of biases, of shape (4H,) """ dx, dh, dc, dWx, dWh, db = None, None, None, None, None, None ############################################################################# # TODO: Implement the backward pass for a single timestep of an LSTM. # # # # HINT: For sigmoid and tanh you can compute local derivatives in terms of # # the output value from the nonlinearity. # ############################################################################# x,f, i, g, o, next_c, next_h, prev_h, Wx, Wh,prev_c = cache N, H = next_c.shape N, D = x.shape dprev_c = dnext_c*f + o*(1-tanh(next_c)**2)*dnext_h*f dnextc_da1 = (1-i)*i*g dnextc_da2 = prev_c*(1-f)*f dnextc_da3 = 0 dnextc_da4 = i*(1-g**2) dnexth_da1 = o*(1-tanh(next_c)**2)*dnextc_da1 dnexth_da2 = o*(1-tanh(next_c)**2)*dnextc_da2 dnexth_da3 = (1-o)*o*tanh(next_c)+o*(1-tanh(next_c)**2)*dnextc_da3 dnexth_da4 = o*(1-tanh(next_c)**2)*dnextc_da4 temp = np.hstack((dnextc_da1*dnext_c+dnexth_da1*dnext_h, \ dnextc_da2*dnext_c+dnexth_da2*dnext_h,\ dnextc_da3*dnext_c+dnexth_da3*dnext_h, \ dnextc_da4*dnext_c+dnexth_da4*dnext_h)) dWx = np.dot(x.T, temp) dWh = np.dot(prev_h.T, temp) db = np.sum(temp, axis=0) dx = np.dot(temp, Wx.T) dprev_h = np.dot(temp, Wh.T) ############################################################################## # END OF YOUR CODE # ############################################################################## return dx, dprev_h, dprev_c, dWx, dWh, dbdef lstm_forward(x, h0, Wx, Wh, b): """ Forward pass for an LSTM over an entire sequence of data. We assume an input sequence composed of T vectors, each of dimension D. The LSTM uses a hidden size of H, and we work over a minibatch containing N sequences. After running the LSTM forward, we return the hidden states for all timesteps. Note that the initial cell state is passed as input, but the initial cell state is set to zero. Also note that the cell state is not returned; it is an internal variable to the LSTM and is not accessed from outside. Inputs: - x: Input data of shape (N, T, D) - h0: Initial hidden state of shape (N, H) - Wx: Weights for input-to-hidden connections, of shape (D, 4H) - Wh: Weights for hidden-to-hidden connections, of shape (H, 4H) - b: Biases of shape (4H,) Returns a tuple of: - h: Hidden states for all timesteps of all sequences, of shape (N, T, H) - cache: Values needed for the backward pass. """ h, cache = None, None ############################################################################# # TODO: Implement the forward pass for an LSTM over an entire timeseries. # # You should use the lstm_step_forward function that you just defined. # ############################################################################# N, T, D = x.shape N, H = h0.shape h = np.zeros((N, T, H)) next_ct = 0 next_ht = h0 cache = [] for t in range(T): next_ht, next_ct, cache_t = lstm_step_forward(x[:,t,:], next_ht, next_ct, Wx, Wh, b) h[:,t,:] = next_ht.copy() cache.append(cache_t) ############################################################################## # END OF YOUR CODE # ############################################################################## return h, cachedef lstm_backward(dh, cache): """ Backward pass for an LSTM over an entire sequence of data.] Inputs: - dh: Upstream gradients of hidden states, of shape (N, T, H) - cache: Values from the forward pass Returns a tuple of: - dx: Gradient of input data of shape (N, T, D) - dh0: Gradient of initial hidden state of shape (N, H) - dWx: Gradient of input-to-hidden weight matrix of shape (D, 4H) - dWh: Gradient of hidden-to-hidden weight matrix of shape (H, 4H) - db: Gradient of biases, of shape (4H,) """ dx, dh0, dWx, dWh, db = None, None, None, None, None ############################################################################# # TODO: Implement the backward pass for an LSTM over an entire timeseries. # # You should use the lstm_step_backward function that you just defined. # ############################################################################# N, T, H = dh.shape _,D = cache[0][0].shape dx = np.zeros((N, T, D)) dh0 = np.zeros((N, H)) dWx = np.zeros((D, 4*H)) dWh = np.zeros((H, 4*H)) db = np.zeros((4*H)) dprev_ct = 0 dprev_ht = 0 for t in reversed(range(T)): o, next_c= cache[t][4:6] dx_t, dprev_ht, dprev_ct, dWx_t, dWh_t, db_t = \ lstm_step_backward(dh[:,t,:]+dprev_ht, dprev_ct, \ cache[t]) dx[:,t,:] = dx_t dWx += dWx_t dWh += dWh_t db += db_t dh0 = dprev_ht ############################################################################## # END OF YOUR CODE # ############################################################################## return dx, dh0, dWx, dWh, dbdef temporal_affine_forward(x, w, b): """ Forward pass for a temporal affine layer. The input is a set of D-dimensional vectors arranged into a minibatch of N timeseries, each of length T. We use an affine function to transform each of those vectors into a new vector of dimension M. Inputs: - x: Input data of shape (N, T, D) - w: Weights of shape (D, M) - b: Biases of shape (M,) Returns a tuple of: - out: Output data of shape (N, T, M) - cache: Values needed for the backward pass """ N, T, D = x.shape M = b.shape[0] out = x.reshape(N * T, D).dot(w).reshape(N, T, M) + b cache = x, w, b, out return out, cachedef temporal_affine_backward(dout, cache): """ Backward pass for temporal affine layer. Input: - dout: Upstream gradients of shape (N, T, M) - cache: Values from forward pass Returns a tuple of: - dx: Gradient of input, of shape (N, T, D) - dw: Gradient of weights, of shape (D, M) - db: Gradient of biases, of shape (M,) """ x, w, b, out = cache N, T, D = x.shape M = b.shape[0] dx = dout.reshape(N * T, M).dot(w.T).reshape(N, T, D) dw = dout.reshape(N * T, M).T.dot(x.reshape(N * T, D)).T db = dout.sum(axis=(0, 1)) return dx, dw, dbdef temporal_softmax_loss(x, y, mask, verbose=False): """ A temporal version of softmax loss for use in RNNs. We assume that we are making predictions over a vocabulary of size V for each timestep of a timeseries of length T, over a minibatch of size N. The input x gives scores for all vocabulary elements at all timesteps, and y gives the indices of the ground-truth element at each timestep. We use a cross-entropy loss at each timestep, summing the loss over all timesteps and averaging across the minibatch. As an additional complication, we may want to ignore the model output at some timesteps, since sequences of different length may have been combined into a minibatch and padded with NULL tokens. The optional mask argument tells us which elements should contribute to the loss. Inputs: - x: Input scores, of shape (N, T, V) - y: Ground-truth indices, of shape (N, T) where each element is in the range 0 <= y[i, t] < V - mask: Boolean array of shape (N, T) where mask[i, t] tells whether or not the scores at x[i, t] should contribute to the loss. Returns a tuple of: - loss: Scalar giving loss - dx: Gradient of loss with respect to scores x. """ N, T, V = x.shape x_flat = x.reshape(N * T, V) y_flat = y.reshape(N * T) mask_flat = mask.reshape(N * T) probs = np.exp(x_flat - np.max(x_flat, axis=1, keepdims=True)) probs /= np.sum(probs, axis=1, keepdims=True) loss = -np.sum(mask_flat * np.log(probs[np.arange(N * T), y_flat])) / N dx_flat = probs.copy() dx_flat[np.arange(N * T), y_flat] -= 1 dx_flat /= N dx_flat *= mask_flat[:, None] if verbose: print 'dx_flat: ', dx_flat.shape dx = dx_flat.reshape(N, T, V) return loss, dx
0 0
- 循环神经网络
- 循环神经网络
- 循环神经网络
- 循环神经网络
- 循环神经网络
- 循环神经网络
- 循环神经网络
- 循环神经网络
- 循环神经网络
- 循环神经网络源码剖析
- 循环神经网络(RNN)
- 循环神经网络(RNN)
- 循环神经网络(一)
- 循环神经网络(二)
- 循环神经网络与LSTM
- 循环神经网络RNN简介
- tensorflow 循环神经网络RNN
- Tensorflow创建循环神经网络
- Android、IOS 图片上传接口(Java实现Servlet)
- Java 各种概念总结--持续更新中
- angular.js学习笔记
- tensorflow模块因缺少库加载不成功
- 如何在Java中导入Excel表数据
- 循环神经网络
- android activity的关闭处理的几种方式
- stm32 F407串口通信出现乱码
- seaj入门
- 给QLineEdit加上一个搜索按钮
- 【个人笔记重点,不作为参考】主题:升级nodejs版本
- EXCEL:统计某个字符串中,某字符或某字符串出现的次数
- matlab与C语言混合编程-----利用Mex函数
- System.out.println和System.err.println的区别