Softmax with cross-entropy derivative

softmax

对于向量$Z=(z_1,z_2,…,z_C)$，计算它对应的softmax向量$Y=(y_1,y_2,…,y_C)$:

$$y_c = \varsigma(Z)_c = \frac{e^{z_c}}{\sum_{d=1}^C e^{z_d}} ,c=1,2,...,C \tag{1}$$

分母$\sum_{d=1}^C e^{z_d}$为归一化因子，使得$\sum_{c=1}^C y_c = 1$。

def softmax(z):
  '''softmax function'''
  return np.exp(z)/np.sum(np.exp(z))

softmax 的梯度

求梯度$\frac{\partial{y_i}}{\partial{z_j}}$

令$\sum_C = \sum_{d=1}^C e^{z_d}$，那么对于公式(1)可简写成$y_c = \frac{e^{z_c}}{\sum_C}$。

$$(e^x)^{'} = e^x$$ $$\left(\frac{1}{x}\right)^{'}= -\frac{1}{x^2}$$

当i=j时，

$$\begin{aligned} \frac{\partial{y_i}}{\partial{z_j}} = \frac{\partial{y_i}}{\partial{z_i}} &=\frac{\partial{ \frac{e^{z_i}}{\sum_C}}}{\partial{z_i}}\\ &= e^{z_i}\frac{-e^{z_i}}{\sum_C^2}+\frac{e^{z_i}}{\sum_C}\\ &= \frac{e^{z_i}\sum_C-e^{z_i}e^{z_i}}{\sum_C^2}\\ &= \frac{e^{z_i}}{\sum_C}\frac{\sum_C-e^{z_i}}{\sum_C}\\ &= \frac{e^{z_i}}{\sum_C}(1-\frac{e^{z_i}}{\sum_C})\\ &= y_i(1-y_i) \end{aligned}\tag{2}$$

当i!=j时，

$$\begin{aligned} \frac{\partial{y_i}}{\partial{z_j}} &=\frac{\partial{ \frac{e^{z_i}}{\sum_C}}}{\partial{z_j}}\\ &= e^{z_i}\frac{-e^{z_j}}{\sum_C^2}+\frac{0}{\sum_C}\\ &= -\frac{e^{z_i}}{\sum_C}\frac{e^{z_j}}{\sum_C}\\ &= -y_iy_j \end{aligned}\tag{3}$$

cross-entropy loss function

从最大似然估计推出交叉熵损失函数

• 最大似然估计函数

  $$\mathrm{argmax}_{\theta} \mathcal{L}(\theta|t,z) \tag{4}$$

• 似然函数写成联合概率分布形式

  $$P(t,z|\theta) = P(t|z,\theta)P(z|\theta) \tag{5}$$

• 对于给定参数，式(5)可简化成

  $$P(t|z) = \prod_{i=c}^C P(t_c|z)^{t_c} = \prod_{i=c}^C \varsigma(z)_c^{t_c} = \prod_{i=c}^C y_c^{t_c} \tag{6}$$

• 那么，对数似然函数即可简化成:

  $$-\log \mathcal{L}(\theta|t,z) =\xi(t,z)= -\log \prod_{i=c}^C y_c^{t_c} = \sum_{i=c}^C t_c.\log(y_c) \tag{7}$$

• 对于多个样本，对应的对数似然函数为:

  $$\xi(T,Y) = \sum_{i=1}^n \xi(t_i,y_i) = -\sum_{i=1}^n \sum_{c=1}^C t_{ic}\log(y_{ic}) \tag{8}$$

交叉熵的梯度

对公式(7)求关于z的梯度：

以e为底的对数导数:$(\log x)^{'}=\frac{1}{x}$

对于t向量是one-hot向量，有$\sum_{j=1}^C t_j=1$

$$\begin{aligned} \frac{\partial{\xi}}{\partial{z_i}} &= -\sum_{j=1}^C \frac{\partial t_j \log(y_j)}{\partial z_i} = -\sum_{j=1}^C t_j \frac{\partial\log(y_j)}{\partial{z_i}} = -\sum_{j=1}^C t_j\frac{1}{y_j}\frac{\partial{y_j}}{\partial{z_i}}\\ &= -\frac{t_i}{y_i}\frac{\partial{y_j}}{\partial{z_i}} - \sum_{j\ne i}^C \frac{t_j}{y_j}\frac{\partial{y_j}}{\partial{z_i}}\\ &= -\frac{t_i}{y_i}y_i(1-y_i) - \sum_{j\ne i}^C \frac{t_j}{y_j}(-y_jy_i)\\ &= -t_i+t_iy_i+\sum_{j\ne i}^Ct_jy_i\\ &= -t_i +\sum_{j=1}^Ct_jy_i\\ &=-t_i+y_i\sum_{j=1}^C t_j\\ &=y_i-t_i \end{aligned}\tag{9}$$

参考资料

• Softmax Regression

• Softmax classification with cross-entropy