矩阵求导（Matrix Derivative）也称作矩阵微分（Matrix Differential），在机器学习、图像处理、最优化等领域的公式推导中经常用到。矩阵求导实际上是多元变量的微积分问题，只是应用在矩阵空间上而已，即为标量求导的一个推广，他的定义为将自变量中的每一个数与因变量中的每一个数求导。
具体地，假设存在 A m × n A_{m \times n} A m × n B p × q B_{p \times q} B p × q ∂ A ∂ B \frac{\partial A}{\partial B} ∂ B ∂ A A A A B B B m × n × p × q m \times n \times p \times q m × n × p × q m × ( n × p × q ) m \times (n \times p \times q) m × ( n × p × q ) ( m × n × p ) × q (m \times n \times p) \times q ( m × n × p ) × q
以分布布局为例子，一共有以下几个矩阵求导法则。分母布局是什么意思呢？简单的说就是以分母为一个基准，希望求导出来的结果和分母的维度相同。除了分母布局以外还有分子布局。分子布局和分母布局的求导结果通常相差一个转置。
基本法则#
法则 0 ：标量对标量求导#
略。详细的请参考高等数学。
法则 1 ：标量对向量求导#
考虑我们有 f f f x = [ x 1 x 2 ⋯ x p ] T x = \begin{bmatrix} x_1 & x_2 & \cdots & x_p \end{bmatrix}^{T} x = [ x 1 x 2 ⋯ x p ] T p × 1 p \times 1 p × 1
∂ f ∂ x = [ ∂ f ∂ x 1 ∂ f ∂ x 2 ⋯ ∂ f ∂ x p ] T \frac{\partial f}{\partial x}=\begin{bmatrix}\frac{\partial f}{\partial x_1} & \frac{\partial f}{\partial x_2} & \cdots & \frac{\partial f}{\partial x_p}\end{bmatrix}^{T} ∂ x ∂ f = [ ∂ x 1 ∂ f ∂ x 2 ∂ f ⋯ ∂ x p ∂ f ] T
可以看得出，求导出来的结果维度是和分母 x x x x x x
法则 2 ：向量对标量求导#
考虑我们有 f = [ f 1 f 2 ⋯ f m ] T f = \begin{bmatrix} f_1 & f_2 & \cdots & f_m \end{bmatrix}^{T} f = [ f 1 f 2 ⋯ f m ] T m × 1 m \times 1 m × 1 x x x
∂ f ∂ x = [ ∂ f 1 ∂ x ∂ f 2 ∂ x ⋯ ∂ f m ∂ x ] \frac{\partial f}{\partial x}=\begin{bmatrix}\frac{\partial f_1}{\partial x} & \frac{\partial f_2}{\partial x} & \cdots & \frac{\partial f_m}{\partial x}\end{bmatrix} ∂ x ∂ f = [ ∂ x ∂ f 1 ∂ x ∂ f 2 ⋯ ∂ x ∂ f m ]
可以看得出，这个时候求导出来的结果维度和分子 f f f f f f
法则 3 ：向量对向量求导#
考虑我们有 f = [ f 1 f 2 ⋯ f m ] T f = \begin{bmatrix} f_1 & f_2 & \cdots & f_m \end{bmatrix}^{T} f = [ f 1 f 2 ⋯ f m ] T m × 1 m \times 1 m × 1 x = [ x 1 x 2 ⋯ x p ] T x = \begin{bmatrix} x_1 & x_2 & \cdots & x_p \end{bmatrix}^{T} x = [ x 1 x 2 ⋯ x p ] T p × 1 p \times 1 p × 1
∂ f ∂ x = [ ∂ f 1 ∂ x 1 ∂ f 2 ∂ x 1 ⋯ ∂ f m ∂ x 1 ∂ f 1 ∂ x 2 ∂ f 2 ∂ x 2 ⋯ ∂ f m ∂ x 2 ⋮ ⋮ ⋱ ⋮ ∂ f 1 ∂ x p ∂ f 2 ∂ x p ⋯ ∂ f m ∂ x p ] \frac{\partial f}{\partial x}=\begin{bmatrix}\frac{\partial f_1}{\partial x_1} & \frac{\partial f_2}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_1} \\ \frac{\partial f_1}{\partial x_2} & \frac{\partial f_2}{\partial x_2} & \cdots & \frac{\partial f_m}{\partial x_2} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial f_1}{\partial x_p} & \frac{\partial f_2}{\partial x_p} & \cdots & \frac{\partial f_m}{\partial x_p} \end{bmatrix} ∂ x ∂ f = ∂ x 1 ∂ f 1 ∂ x 2 ∂ f 1 ⋮ ∂ x p ∂ f 1 ∂ x 1 ∂ f 2 ∂ x 2 ∂ f 2 ⋮ ∂ x p ∂ f 2 ⋯ ⋯ ⋱ ⋯ ∂ x 1 ∂ f m ∂ x 2 ∂ f m ⋮ ∂ x p ∂ f m
这时求导结果的维度为 p × m p \times m p × m
法则 4 ：标量对矩阵求导#
考虑我们有 f f f x p × q x_{p \times q} x p × q
∂ f ∂ x = [ ∂ f ∂ x 11 ∂ f ∂ x 12 ⋯ ∂ f ∂ x 1 q ∂ f ∂ x 21 ∂ f ∂ x 22 ⋯ ∂ f ∂ x 2 q ⋮ ⋮ ⋱ ⋮ ∂ f ∂ x p 1 ∂ f ∂ x p 2 ⋯ ∂ f ∂ x p q ] \frac{\partial f}{\partial x}=\begin{bmatrix}\frac{\partial f}{\partial x_{11}} & \frac{\partial f}{\partial x_{12}} & \cdots & \frac{\partial f}{\partial x_{1q}} \\ \frac{\partial f}{\partial x_{21}} & \frac{\partial f}{\partial x_{22}} & \cdots & \frac{\partial f}{\partial x_{2q}} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial f}{\partial x_{p1}} & \frac{\partial f}{\partial x_{p2}} & \cdots & \frac{\partial f}{\partial x_{pq}} \end{bmatrix} ∂ x ∂ f = ∂ x 11 ∂ f ∂ x 21 ∂ f ⋮ ∂ x p 1 ∂ f ∂ x 12 ∂ f ∂ x 22 ∂ f ⋮ ∂ x p 2 ∂ f ⋯ ⋯ ⋱ ⋯ ∂ x 1 q ∂ f ∂ x 2 q ∂ f ⋮ ∂ x pq ∂ f
同样，我们求导结果和分母 x x x p × q p \times q p × q
法则 5 ：矩阵对向量求导#
考虑我们有 f m × n f_{m \times n} f m × n x x x
∂ f ∂ x = [ ∂ f 11 ∂ x ∂ f 21 ∂ x ⋯ ∂ f m 1 ∂ x ∂ f 21 ∂ x ∂ f 22 ∂ x ⋯ ∂ f m 2 ∂ x 2 q ⋮ ⋮ ⋱ ⋮ ∂ f n 1 ∂ x ∂ f n 2 ∂ x ⋯ ∂ f n m ∂ x ] \frac{\partial f}{\partial x}=\begin{bmatrix}\frac{\partial f_{11}}{\partial x} & \frac{\partial f_{21}}{\partial x} & \cdots & \frac{\partial f_{m1}}{\partial x} \\ \frac{\partial f_{21}}{\partial x} & \frac{\partial f_{22}}{\partial x} & \cdots & \frac{\partial f_{m2}}{\partial x_{2q}} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial f_{n1}}{\partial x} & \frac{\partial f_{n2}}{\partial x} & \cdots & \frac{\partial f_{nm}}{\partial x} \end{bmatrix} ∂ x ∂ f = ∂ x ∂ f 11 ∂ x ∂ f 21 ⋮ ∂ x ∂ f n 1 ∂ x ∂ f 21 ∂ x ∂ f 22 ⋮ ∂ x ∂ f n 2 ⋯ ⋯ ⋱ ⋯ ∂ x ∂ f m 1 ∂ x 2 q ∂ f m 2 ⋮ ∂ x ∂ f nm
我们求导的结果与分子相反，为 n × m n \times m n × m
其余：向量与矩阵之间以及矩阵与矩阵之间的求导#
当我们的自变量与因变量都为不为标量时，根据我们对矩阵求导实质的讨论，势必会得出大量的导数难以被排列。例如，一般情况下，假设我们有 f m × n f_{m \times n} f m × n x p × q x_{p \times q} x p × q m × n × p × q m \times n \times p \times q m × n × p × q
第一种定义#
我们按照之前的法则，将 ∂ f ∂ x \frac{\partial f}{\partial x} ∂ x ∂ f f f f x x x f f f ∂ f i j ∂ x \frac{\partial f_{ij}}{\partial x} ∂ x ∂ f ij f i j f_{ij} f ij m p × n q mp \times nq m p × n q
第二种定义（主流）#
这种定义是将矩阵对矩阵求导问题归约到向量对向量求导。即对矩阵先做向量化处理，然后再求导：
∂ f ∂ x = ∂ v e c ( f ) ∂ v e c ( x ) \frac{\partial f}{\partial x}=\frac{\partial vec(f)}{\partial vec(x)} ∂ x ∂ f = ∂ v ec ( x ) ∂ v ec ( f )
其中，向量化的实现方法分为列向量化和行向量化。我们以列向量化为例，将 f m × n f_{m \times n} f m × n x p × q x_{p \times q} x p × q f m n × 1 f_{mn \times 1} f mn × 1 x p q × 1 x_{pq \times 1} x pq × 1 p q × m n pq \times mn pq × mn
有用的公式#
下列公式中，A m × 1 A_{m \times 1} A m × 1 x m × 1 x_{m \times 1} x m × 1 B m × m B_{m \times m} B m × m
编号 公式 1 ∂ x T A ∂ x = ∂ A T x ∂ x = A \frac{\partial{x^{T}A}}{\partial{x}} = \frac{\partial{A^{T}x}}{\partial{x}} = A ∂ x ∂ x T A = ∂ x ∂ A T x = A 2 ∂ x T x ∂ x = x \frac{\partial{x^{T}x}}{\partial{x}} = x ∂ x ∂ x T x = x 3 ∂ x T B x ∂ x = ( B + B T ) x \frac{\partial{x^{T}Bx}}{\partial{x}} = (B + B^{T})x ∂ x ∂ x T B x = ( B + B T ) x
下列公式是一些关于矩阵迹的公式。其中， a a a A A A B B B C C C
编号 公式 1 t r ( a ) = a tr(a) = a t r ( a ) = a 2 t r ( A ) = t r ( A T ) tr(A) = tr(A^T) t r ( A ) = t r ( A T ) 3 t r ( A B ) = t r ( B A ) tr(AB) = tr(BA) t r ( A B ) = t r ( B A ) 4 t r ( A B C ) = t r ( C A B ) = t r ( B C A ) tr(ABC) = tr(CAB) = tr(BCA) t r ( A BC ) = t r ( C A B ) = t r ( BC A ) 5 ∂ t r ( A B ) ∂ A = B T \frac{\partial{tr(AB)}}{\partial{A}} = B^T ∂ A ∂ t r ( A B ) = B T 6 ∂ t r ( A B A T C ) ∂ A = C A B + C T A B T \frac{\partial{tr(ABA^{T}C)}}{\partial{A}} = CAB + C^{T}AB^{T} ∂ A ∂ t r ( A B A T C ) = C A B + C T A B T
一些公式的证明#
令：
A m × 1 = [ A 1 A 2 ⋯ A m ] T A_{m \times 1} = \begin{bmatrix} A_1 & A_2 & \cdots & A_m \end{bmatrix} ^ {T} A m × 1 = [ A 1 A 2 ⋯ A m ] T
B m × m = B m × m = [ B 11 B 12 ⋯ B 1 m B 21 B 22 ⋯ B 2 m ⋮ ⋮ ⋱ ⋮ B m 1 B m 2 ⋯ B m m ] B_{m \times m} = B_{m \times m} = \begin{bmatrix} B_{11} & B_{12} & \cdots & B_{1m} \\ B_{21} & B_{22} & \cdots & B_{2m} \\ \vdots & \vdots & \ddots & \vdots \\ B_{m1} & B_{m2} & \cdots & B_{mm} \end{bmatrix} B m × m = B m × m = B 11 B 21 ⋮ B m 1 B 12 B 22 ⋮ B m 2 ⋯ ⋯ ⋱ ⋯ B 1 m B 2 m ⋮ B mm
x m × 1 = [ x 1 x 2 ⋯ x m ] T x_{m \times 1} = \begin{bmatrix} x_1 & x_2 & \cdots & x_m \end{bmatrix} ^ {T} x m × 1 = [ x 1 x 2 ⋯ x m ] T
公式 1#
∂ x T A ∂ x = ∂ A T x ∂ x = A \frac{\partial{x^{T}A}}{\partial{x}} = \frac{\partial{A^{T}x}}{\partial{x}} = A ∂ x ∂ x T A = ∂ x ∂ A T x = A
因为 A m × 1 A_{m \times 1} A m × 1 x m × 1 x_{m \times 1} x m × 1 x T A = A T x = ∑ i = 1 m A i x i x^{T}A = A^{T}x = \sum_{i=1}^{m}{A_{i}x_{i}} x T A = A T x = ∑ i = 1 m A i x i
∂ x T A ∂ x = ∂ A T x ∂ x \frac{\partial{x^{T}A}}{\partial{x}} = \frac{\partial{A^{T}x}}{\partial{x}} ∂ x ∂ x T A = ∂ x ∂ A T x
= [ ∂ ∑ i = 1 m A i x i ∂ x 1 ∂ ∑ i = 1 m A i x i ∂ x 2 ⋯ ∂ ∑ i = 1 m A i x i ∂ x m ] = \begin{bmatrix} \frac{\partial{\sum_{i=1}^{m}{A_{i}x_{i}}}}{\partial{x_1}} \\ \frac{\partial{\sum_{i=1}^{m}{A_{i}x_{i}}}}{\partial{x_2}} \\ \cdots \\ \frac{\partial{\sum_{i=1}^{m}{A_{i}x_{i}}}}{\partial{x_m}} \end{bmatrix} = ∂ x 1 ∂ ∑ i = 1 m A i x i ∂ x 2 ∂ ∑ i = 1 m A i x i ⋯ ∂ x m ∂ ∑ i = 1 m A i x i
= [ A 1 A 2 ⋯ A m ] = \begin{bmatrix} A_1 \\ A_2 \\ \cdots \\ A_m \end{bmatrix} = A 1 A 2 ⋯ A m
= A = A = A
公式 2#
同理 公式 1
公式 3#
∂ x T B x ∂ x = ( B + B T ) x \frac{\partial{x^{T}Bx}}{\partial{x}} = (B + B^{T})x ∂ x ∂ x T B x = ( B + B T ) x
由题意可得， x T B x x^{T}Bx x T B x
∂ x T B x ∂ x \frac{\partial{x^{T}Bx}}{\partial{x}} ∂ x ∂ x T B x
= [ ∂ ∑ i = 1 m ∑ j = 1 m B i j x i x j ∂ x 1 ∂ ∑ i = 1 m ∑ j = 1 m B i j x i x j ∂ x 2 ⋯ ∂ ∑ i = 1 m ∑ j = 1 m B i j x i x j ∂ x m ] = \begin{bmatrix} \frac{\partial{\sum_{i=1}^{m}{\sum_{j=1}^{m}{B_{ij}x_{i}x_{j}}}}}{\partial{x_1}} \\ \frac{\partial{\sum_{i=1}^{m}{\sum_{j=1}^{m}{B_{ij}x_{i}x_{j}}}}}{\partial{x_2}} \\ \cdots \\ \frac{\partial{\sum_{i=1}^{m}{\sum_{j=1}^{m}{B_{ij}x_{i}x_{j}}}}}{\partial{x_m}} \end{bmatrix} = ∂ x 1 ∂ ∑ i = 1 m ∑ j = 1 m B ij x i x j ∂ x 2 ∂ ∑ i = 1 m ∑ j = 1 m B ij x i x j ⋯ ∂ x m ∂ ∑ i = 1 m ∑ j = 1 m B ij x i x j
由导数法则有：
∂ f ( x ) g ( x ) ∂ x = ∂ f ( x ) x g ( x ) + f ( x ) ∂ g ( x ) ∂ x \frac{\partial{f(x)g(x)}}{\partial{x}} = \frac{\partial{f(x)}}{x}g(x) + f(x)\frac{\partial{g(x)}}{\partial{x}} ∂ x ∂ f ( x ) g ( x ) = x ∂ f ( x ) g ( x ) + f ( x ) ∂ x ∂ g ( x )
于是，原式继续有：
= [ ∑ i = 1 m B i 1 x i + ∑ j = 1 m B 1 j x j ∑ i = 1 m B i 2 x i + ∑ j = 1 m B 2 j x j ⋯ ∑ i = 1 m B i m x i + ∑ j = 1 m B m j x j ] = \begin{bmatrix} \sum_{i=1}^{m}{B_{i1}x_i} + \sum_{j=1}^{m}{B_{1j}x_j} \\ \sum_{i=1}^{m}{B_{i2}x_i} + \sum_{j=1}^{m}{B_{2j}x_j} \\ \cdots \\ \sum_{i=1}^{m}{B_{im}x_i} + \sum_{j=1}^{m}{B_{mj}x_j} \end{bmatrix} = ∑ i = 1 m B i 1 x i + ∑ j = 1 m B 1 j x j ∑ i = 1 m B i 2 x i + ∑ j = 1 m B 2 j x j ⋯ ∑ i = 1 m B im x i + ∑ j = 1 m B mj x j
= [ ∑ i = 1 m B i 1 x i ∑ i = 1 m B i 2 x i ⋯ ∑ i = 1 m B i m x i ] + [ ∑ j = 1 m B 1 j x j ∑ j = 1 m B 2 j x j ⋯ ∑ j = 1 m B m j x j ] = \begin{bmatrix} \sum_{i=1}^{m}{B_{i1}x_i} \\ \sum_{i=1}^{m}{B_{i2}x_i} \\ \cdots \\ \sum_{i=1}^{m}{B_{im}x_i} \end{bmatrix} + \begin{bmatrix} \sum_{j=1}^{m}{B_{1j}x_j} \\ \sum_{j=1}^{m}{B_{2j}x_j} \\ \cdots \\ \sum_{j=1}^{m}{B_{mj}x_j} \end{bmatrix} = ∑ i = 1 m B i 1 x i ∑ i = 1 m B i 2 x i ⋯ ∑ i = 1 m B im x i + ∑ j = 1 m B 1 j x j ∑ j = 1 m B 2 j x j ⋯ ∑ j = 1 m B mj x j
= [ B 11 B 21 ⋯ B m 1 B 12 B 22 ⋯ B m 2 ⋮ ⋮ ⋱ ⋮ B 1 m B 2 m ⋯ B m m ] [ x 1 x 2 ⋮ x m ] + [ B 11 B 12 ⋯ B 1 m B 21 B 22 ⋯ B 2 m ⋮ ⋮ ⋱ ⋮ B m 1 B m 2 ⋯ B m m ] [ x 1 x 2 ⋮ x m ] = \begin{bmatrix} B_{11} & B{21} & \cdots & B{m1} \\ B_{12} & B{22} & \cdots & B{m2} \\ \vdots & \vdots & \ddots & \vdots \\ B_{1m} & B{2m} & \cdots & B{mm} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_m \end{bmatrix}+ \begin{bmatrix} B_{11} & B{12} & \cdots & B{1m} \\ B_{21} & B{22} & \cdots & B{2m} \\ \vdots & \vdots & \ddots & \vdots \\ B_{m1} & B{m2} & \cdots & B{mm} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_m \end{bmatrix} = B 11 B 12 ⋮ B 1 m B 21 B 22 ⋮ B 2 m ⋯ ⋯ ⋱ ⋯ B m 1 B m 2 ⋮ B mm x 1 x 2 ⋮ x m + B 11 B 21 ⋮ B m 1 B 12 B 22 ⋮ B m 2 ⋯ ⋯ ⋱ ⋯ B 1 m B 2 m ⋮ B mm x 1 x 2 ⋮ x m
= ( A T + A ) x = ( A + A T ) x = (A^{T} + A)x = (A + A^{T})x = ( A T + A ) x = ( A + A T ) x
