LSTM模型结构分解

forget gate

$$f_t = \mathrm{sigmoid}(x_tU^f+h_{t-1}W^f) \tag{1}$$

input gate

$$i_t = \mathrm{sigmoid}(x_tU^i+h_{t-1}W^i) \tag{2}$$ $$\hat{c_t} = \mathrm{tanh}(x_tU^g+h_{t-1}W^g) \tag{3}$$ $$\textrm{input_gate_output} = i_t \odot \hat{c_t} \tag{4}$$

output gate

$$o_t = \mathrm{sigmoid}(x_tU^o+h_{t-1}W^o) \tag{5}$$ $$h_t = o_t \odot \mathrm{tanh}(c_t) \tag{6}$$

cell state

$$c_t = c_{t-1}\odot f_t + \textrm{input_gate_output} \tag{7}$$

实现

(略)

其他

tanh 激活函数的作用

• tanh squishes value between -1 and 1

  

• vector transformations without tanh

  

• vector transformations with tanh

  

sigmoid激活函数的作用

当向量通过sigmoid激活函数后，可以使得输出保持在[0,1]之间的范围里，越接近0表示分量贡献越小，接近1表示分量贡献越大。起到mask的作用。

• Sigmoid squishes values between 0 and 1

  

参考资料

• Illustrated Guide to LSTM’s and GRU’s: A step by step explanation
• Recurrent Neural Network Tutorial, Part 4 – Implementing a GRU/LSTM RNN with Python and Theano
• Animated RNN, LSTM and GRU
• Vanilla LSTM with numpy