GRU模型结构分解

Update Gate

update gate 相当于lstm中的forget gate和input gate，用于决定哪些旧的信息该遗忘，哪些新的信息该记住。

$$z = \mathrm{sigmoid}(x_tU^z+h_{t-1}W^z) \tag{1}$$ $$\textrm{update_gate_output} = 1-z \tag{2}$$

Reset Gate

用于决定应该忘记多少过去的信息。

$$r = \mathrm{sigmoid}(x_tU^r+h_{t-1}W^r) \tag{3}$$

Output Gate

$$\textrm{output_gate_output} = \mathrm{tanh}(x_tU^o+(h_{t-1}\odot r)W^o) \tag{4}$$ $$h_t = \textrm{update_gate_output}\odot h_{t-1}+\textrm{output_gate_output}\odot z\tag{5}$$

代码实现

(略)

参考资料

• Animated RNN, LSTM and GRU
• Illustrated Guide to LSTM’s and GRU’s: A step by step explanation
• An Empirical Exploration of Recurrent Network Architectures
• rnn-tutorial-gru-lstm