矩陣求導（Matrix Derivative）也稱作矩陣微分（Matrix Differential），在機器學習、圖像處理、最優化等領域的公式推導中經常用到。矩陣求導實際上是多元變量的微積分問題，只是應用在矩陣空間上而已，即為標量求導的一個推廣，它的定義為將自變量中的每一個數與因變量中的每一個數求導。
具體地，假設存在 A m × n A_{m \times n} A m × n B p × q B_{p \times q} B p × q ∂ A ∂ B \frac{\partial A}{\partial B} ∂ B ∂ A A A A B B B m × n × p × q m \times n \times p \times q m × n × p × q m × ( n × p × q ) m \times (n \times p \times q) m × ( n × p × q ) ( m × n × p ) × q (m \times n \times p) \times q ( m × n × p ) × q
以分布佈局為例子，一共有以下幾個矩陣求導法則。分母佈局是什麼意思呢？簡單的說就是以分母為一個基準，希望求導出來的結果和分母的維度相同。除了分母佈局以外還有分子佈局。分子佈局和分母佈局的求導結果通常相差一個轉置。
基本法則#
法則 0 ：標量對標量求導#
略。詳細的請參考高等數學。
法則 1 ：標量對向量求導#
考慮我們有 f f f x = [ x 1 x 2 ⋯ x p ] T x = \begin{bmatrix} x_1 & x_2 & \cdots & x_p \end{bmatrix}^{T} x = [ x 1 x 2 ⋯ x p ] T p × 1 p \times 1 p × 1
∂ f ∂ x = [ ∂ f ∂ x 1 ∂ f ∂ x 2 ⋯ ∂ f ∂ x p ] T \frac{\partial f}{\partial x}=\begin{bmatrix}\frac{\partial f}{\partial x_1} & \frac{\partial f}{\partial x_2} & \cdots & \frac{\partial f}{\partial x_p}\end{bmatrix}^{T} ∂ x ∂ f = [ ∂ x 1 ∂ f ∂ x 2 ∂ f ⋯ ∂ x p ∂ f ] T
可以看得出，求導出來的結果維度是和分母 x x x x x x
法則 2 ：向量對標量求導#
考慮我們有 f = [ f 1 f 2 ⋯ f m ] T f = \begin{bmatrix} f_1 & f_2 & \cdots & f_m \end{bmatrix}^{T} f = [ f 1 f 2 ⋯ f m ] T m × 1 m \times 1 m × 1 x x x
∂ f ∂ x = [ ∂ f 1 ∂ x ∂ f 2 ∂ x ⋯ ∂ f m ∂ x ] \frac{\partial f}{\partial x}=\begin{bmatrix}\frac{\partial f_1}{\partial x} & \frac{\partial f_2}{\partial x} & \cdots & \frac{\partial f_m}{\partial x}\end{bmatrix} ∂ x ∂ f = [ ∂ x ∂ f 1 ∂ x ∂ f 2 ⋯ ∂ x ∂ f m ]
可以看得出，這個時候求導出來的結果維度和分子 f f f f f f
法則 3 ：向量對向量求導#
考慮我們有 f = [ f 1 f 2 ⋯ f m ] T f = \begin{bmatrix} f_1 & f_2 & \cdots & f_m \end{bmatrix}^{T} f = [ f 1 f 2 ⋯ f m ] T m × 1 m \times 1 m × 1 x = [ x 1 x 2 ⋯ x p ] T x = \begin{bmatrix} x_1 & x_2 & \cdots & x_p \end{bmatrix}^{T} x = [ x 1 x 2 ⋯ x p ] T p × 1 p \times 1 p × 1
∂ f ∂ x = [ ∂ f 1 ∂ x 1 ∂ f 2 ∂ x 1 ⋯ ∂ f m ∂ x 1 ∂ f 1 ∂ x 2 ∂ f 2 ∂ x 2 ⋯ ∂ f m ∂ x 2 ⋮ ⋮ ⋱ ⋮ ∂ f 1 ∂ x p ∂ f 2 ∂ x p ⋯ ∂ f m ∂ x p ] \frac{\partial f}{\partial x}=\begin{bmatrix}\frac{\partial f_1}{\partial x_1} & \frac{\partial f_2}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_1} \\ \frac{\partial f_1}{\partial x_2} & \frac{\partial f_2}{\partial x_2} & \cdots & \frac{\partial f_m}{\partial x_2} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial f_1}{\partial x_p} & \frac{\partial f_2}{\partial x_p} & \cdots & \frac{\partial f_m}{\partial x_p} \end{bmatrix} ∂ x ∂ f = ∂ x 1 ∂ f 1 ∂ x 2 ∂ f 1 ⋮ ∂ x p ∂ f 1 ∂ x 1 ∂ f 2 ∂ x 2 ∂ f 2 ⋮ ∂ x p ∂ f 2 ⋯ ⋯ ⋱ ⋯ ∂ x 1 ∂ f m ∂ x 2 ∂ f m ⋮ ∂ x p ∂ f m
這時求導結果的維度為 p × m p \times m p × m
法則 4 ：標量對矩陣求導#
考慮我們有 f f f x p × q x_{p \times q} x p × q
∂ f ∂ x = [ ∂ f ∂ x 11 ∂ f ∂ x 12 ⋯ ∂ f ∂ x 1 q ∂ f ∂ x 21 ∂ f ∂ x 22 ⋯ ∂ f ∂ x 2 q ⋮ ⋮ ⋱ ⋮ ∂ f ∂ x p 1 ∂ f ∂ x p 2 ⋯ ∂ f ∂ x p q ] \frac{\partial f}{\partial x}=\begin{bmatrix}\frac{\partial f}{\partial x_{11}} & \frac{\partial f}{\partial x_{12}} & \cdots & \frac{\partial f}{\partial x_{1q}} \\ \frac{\partial f}{\partial x_{21}} & \frac{\partial f}{\partial x_{22}} & \cdots & \frac{\partial f}{\partial x_{2q}} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial f}{\partial x_{p1}} & \frac{\partial f}{\partial x_{p2}} & \cdots & \frac{\partial f}{\partial x_{pq}} \end{bmatrix} ∂ x ∂ f = ∂ x 11 ∂ f ∂ x 21 ∂ f ⋮ ∂ x p 1 ∂ f ∂ x 12 ∂ f ∂ x 22 ∂ f ⋮ ∂ x p 2 ∂ f ⋯ ⋯ ⋱ ⋯ ∂ x 1 q ∂ f ∂ x 2 q ∂ f ⋮ ∂ x pq ∂ f
同樣，我們求導結果和分母 x x x p × q p \times q p × q
法則 5 ：矩陣對向量求導#
考慮我們有 f m × n f_{m \times n} f m × n x x x
∂ f ∂ x = [ ∂ f 11 ∂ x ∂ f 21 ∂ x ⋯ ∂ f m 1 ∂ x ∂ f 21 ∂ x ∂ f 22 ∂ x ⋯ ∂ f m 2 ∂ x 2 q ⋮ ⋮ ⋱ ⋮ ∂ f n 1 ∂ x ∂ f n 2 ∂ x ⋯ ∂ f n m ∂ x ] \frac{\partial f}{\partial x}=\begin{bmatrix}\frac{\partial f_{11}}{\partial x} & \frac{\partial f_{21}}{\partial x} & \cdots & \frac{\partial f_{m1}}{\partial x} \\ \frac{\partial f_{21}}{\partial x} & \frac{\partial f_{22}}{\partial x} & \cdots & \frac{\partial f_{m2}}{\partial x_{2q}} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial f_{n1}}{\partial x} & \frac{\partial f_{n2}}{\partial x} & \cdots & \frac{\partial f_{nm}}{\partial x} \end{bmatrix} ∂ x ∂ f = ∂ x ∂ f 11 ∂ x ∂ f 21 ⋮ ∂ x ∂ f n 1 ∂ x ∂ f 21 ∂ x ∂ f 22 ⋮ ∂ x ∂ f n 2 ⋯ ⋯ ⋱ ⋯ ∂ x ∂ f m 1 ∂ x 2 q ∂ f m 2 ⋮ ∂ x ∂ f nm
我們求導的結果與分子相反，為 n × m n \times m n × m
其餘：向量與矩陣之間以及矩陣與矩陣之間的求導#
當我們的自變量與因變量都不為標量時，根據我們對矩陣求導實質的討論，勢必會得出大量的導數難以被排列。例如，一般情況下，假設我們有 f m × n f_{m \times n} f m × n x p × q x_{p \times q} x p × q m × n × p × q m \times n \times p \times q m × n × p × q
第一種定義#
我們按照之前的法則，將 ∂ f ∂ x \frac{\partial f}{\partial x} ∂ x ∂ f f f f x x x f f f ∂ f i j ∂ x \frac{\partial f_{ij}}{\partial x} ∂ x ∂ f ij f i j f_{ij} f ij m p × n q mp \times nq m p × n q
第二種定義（主流）#
這種定義是將矩陣對矩陣求導問題歸約到向量對向量求導。即對矩陣先做向量化處理，然後再求導：
∂ f ∂ x = ∂ v e c ( f ) ∂ v e c ( x ) \frac{\partial f}{\partial x}=\frac{\partial vec(f)}{\partial vec(x)} ∂ x ∂ f = ∂ v ec ( x ) ∂ v ec ( f )
其中，向量化的實現方法分為列向量化和行向量化。我們以列向量化為例，將 f m × n f_{m \times n} f m × n x p × q x_{p \times q} x p × q f m n × 1 f_{mn \times 1} f mn × 1 x p q × 1 x_{pq \times 1} x pq × 1 p q × m n pq \times mn pq × mn
有用的公式#
下列公式中，A m × 1 A_{m \times 1} A m × 1 x m × 1 x_{m \times 1} x m × 1 B m × m B_{m \times m} B m × m
編號 公式 1 ∂ x T A ∂ x = ∂ A T x ∂ x = A \frac{\partial{x^{T}A}}{\partial{x}} = \frac{\partial{A^{T}x}}{\partial{x}} = A ∂ x ∂ x T A = ∂ x ∂ A T x = A 2 ∂ x T x ∂ x = x \frac{\partial{x^{T}x}}{\partial{x}} = x ∂ x ∂ x T x = x 3 ∂ x T B x ∂ x = ( B + B T ) x \frac{\partial{x^{T}Bx}}{\partial{x}} = (B + B^{T})x ∂ x ∂ x T B x = ( B + B T ) x
下列公式是一些關於矩陣跡的公式。其中， a a a A A A B B B C C C
編號 公式 1 t r ( a ) = a tr(a) = a t r ( a ) = a 2 t r ( A ) = t r ( A T ) tr(A) = tr(A^T) t r ( A ) = t r ( A T ) 3 t r ( A B ) = t r ( B A ) tr(AB) = tr(BA) t r ( A B ) = t r ( B A ) 4 t r ( A B C ) = t r ( C A B ) = t r ( B C A ) tr(ABC) = tr(CAB) = tr(BCA) t r ( A BC ) = t r ( C A B ) = t r ( BC A ) 5 ∂ t r ( A B ) ∂ A = B T \frac{\partial{tr(AB)}}{\partial{A}} = B^T ∂ A ∂ t r ( A B ) = B T 6 ∂ t r ( A B A T C ) ∂ A = C A B + C T A B T \frac{\partial{tr(ABA^{T}C)}}{\partial{A}} = CAB + C^{T}AB^{T} ∂ A ∂ t r ( A B A T C ) = C A B + C T A B T
一些公式的證明#
令：
A m × 1 = [ A 1 A 2 ⋯ A m ] T A_{m \times 1} = \begin{bmatrix} A_1 & A_2 & \cdots & A_m \end{bmatrix} ^ {T} A m × 1 = [ A 1 A 2 ⋯ A m ] T
B m × m = B m × m = [ B 11 B 12 ⋯ B 1 m B 21 B 22 ⋯ B 2 m ⋮ ⋮ ⋱ ⋮ B m 1 B m 2 ⋯ B m m ] B_{m \times m} = B_{m \times m} = \begin{bmatrix} B_{11} & B_{12} & \cdots & B_{1m} \\ B_{21} & B_{22} & \cdots & B_{2m} \\ \vdots & \vdots & \ddots & \vdots \\ B_{m1} & B_{m2} & \cdots & B_{mm} \end{bmatrix} B m × m = B m × m = B 11 B 21 ⋮ B m 1 B 12 B 22 ⋮ B m 2 ⋯ ⋯ ⋱ ⋯ B 1 m B 2 m ⋮ B mm
x m × 1 = [ x 1 x 2 ⋯ x m ] T x_{m \times 1} = \begin{bmatrix} x_1 & x_2 & \cdots & x_m \end{bmatrix} ^ {T} x m × 1 = [ x 1 x 2 ⋯ x m ] T
公式 1#
∂ x T A ∂ x = ∂ A T x ∂ x = A \frac{\partial{x^{T}A}}{\partial{x}} = \frac{\partial{A^{T}x}}{\partial{x}} = A ∂ x ∂ x T A = ∂ x ∂ A T x = A
因為 A m × 1 A_{m \times 1} A m × 1 x m × 1 x_{m \times 1} x m × 1 x T A = A T x = ∑ i = 1 m A i x i x^{T}A = A^{T}x = \sum_{i=1}^{m}{A_{i}x_{i}} x T A = A T x = ∑ i = 1 m A i x i
∂ x T A ∂ x = ∂ A T x ∂ x \frac{\partial{x^{T}A}}{\partial{x}} = \frac{\partial{A^{T}x}}{\partial{x}} ∂ x ∂ x T A = ∂ x ∂ A T x
= [ ∂ ∑ i = 1 m A i x i ∂ x 1 ∂ ∑ i = 1 m A i x i ∂ x 2 ⋯ ∂ ∑ i = 1 m A i x i ∂ x m ] = \begin{bmatrix} \frac{\partial{\sum_{i=1}^{m}{A_{i}x_{i}}}}{\partial{x_1}} \\ \frac{\partial{\sum_{i=1}^{m}{A_{i}x_{i}}}}{\partial{x_2}} \\ \cdots \\ \frac{\partial{\sum_{i=1}^{m}{A_{i}x_{i}}}}{\partial{x_m}} \end{bmatrix} = ∂ x 1 ∂ ∑ i = 1 m A i x i ∂ x 2 ∂ ∑ i = 1 m A i x i ⋯ ∂ x m ∂ ∑ i = 1 m A i x i
= [ A 1 A 2 ⋯ A m ] = \begin{bmatrix} A_1 \\ A_2 \\ \cdots \\ A_m \end{bmatrix} = A 1 A 2 ⋯ A m
= A = A = A
公式 2#
同理 公式 1
公式 3#
∂ x T B x ∂ x = ( B + B T ) x \frac{\partial{x^{T}Bx}}{\partial{x}} = (B + B^{T})x ∂ x ∂ x T B x = ( B + B T ) x
由題意可得， x T B x x^{T}Bx x T B x
∂ x T B x ∂ x \frac{\partial{x^{T}Bx}}{\partial{x}} ∂ x ∂ x T B x
= [ ∂ ∑ i = 1 m ∑ j = 1 m B i j x i x j ∂ x 1 ∂ ∑ i = 1 m ∑ j = 1 m B i j x i x j ∂ x 2 ⋯ ∂ ∑ i = 1 m ∑ j = 1 m B i j x i x j ∂ x m ] = \begin{bmatrix} \frac{\partial{\sum_{i=1}^{m}{\sum_{j=1}^{m}{B_{ij}x_{i}x_{j}}}}}{\partial{x_1}} \\ \frac{\partial{\sum_{i=1}^{m}{\sum_{j=1}^{m}{B_{ij}x_{i}x_{j}}}}}{\partial{x_2}} \\ \cdots \\ \frac{\partial{\sum_{i=1}^{m}{\sum_{j=1}^{m}{B_{ij}x_{i}x_{j}}}}}{\partial{x_m}} \end{bmatrix} = ∂ x 1 ∂ ∑ i = 1 m ∑ j = 1 m B ij x i x j ∂ x 2 ∂ ∑ i = 1 m ∑ j = 1 m B ij x i x j ⋯ ∂ x m ∂ ∑ i = 1 m ∑ j = 1 m B ij x i x j
由導數法則有：
∂ f ( x ) g ( x ) ∂ x = ∂ f ( x ) x g ( x ) + f ( x ) ∂ g ( x ) ∂ x \frac{\partial{f(x)g(x)}}{\partial{x}} = \frac{\partial{f(x)}}{x}g(x) + f(x)\frac{\partial{g(x)}}{\partial{x}} ∂ x ∂ f ( x ) g ( x ) = x ∂ f ( x ) g ( x ) + f ( x ) ∂ x ∂ g ( x )
於是，原式繼續有：
= [ ∑ i = 1 m B i 1 x i + ∑ j = 1 m B 1 j x j ∑ i = 1 m B i 2 x i + ∑ j = 1 m B 2 j x j ⋯ ∑ i = 1 m B i m x i + ∑ j = 1 m B m j x j ] = \begin{bmatrix} \sum_{i=1}^{m}{B_{i1}x_i} + \sum_{j=1}^{m}{B_{1j}x_j} \\ \sum_{i=1}^{m}{B_{i2}x_i} + \sum_{j=1}^{m}{B_{2j}x_j} \\ \cdots \\ \sum_{i=1}^{m}{B_{im}x_i} + \sum_{j=1}^{m}{B_{mj}x_j} \end{bmatrix} = ∑ i = 1 m B i 1 x i + ∑ j = 1 m B 1 j x j ∑ i = 1 m B i 2 x i + ∑ j = 1 m B 2 j x j ⋯ ∑ i = 1 m B im x i + ∑ j = 1 m B mj x j
= [ ∑ i = 1 m B i 1 x i ∑ i = 1 m B i 2 x i ⋯ ∑ i = 1 m B i m x i ] + [ ∑ j = 1 m B 1 j x j ∑ j = 1 m B 2 j x j ⋯ ∑ j = 1 m B m j x j ] = \begin{bmatrix} \sum_{i=1}^{m}{B_{i1}x_i} \\ \sum_{i=1}^{m}{B_{i2}x_i} \\ \cdots \\ \sum_{i=1}^{m}{B_{im}x_i} \end{bmatrix} + \begin{bmatrix} \sum_{j=1}^{m}{B_{1j}x_j} \\ \sum_{j=1}^{m}{B_{2j}x_j} \\ \cdots \\ \sum_{j=1}^{m}{B_{mj}x_j} \end{bmatrix} = ∑ i = 1 m B i 1 x i ∑ i = 1 m B i 2 x i ⋯ ∑ i = 1 m B im x i + ∑ j = 1 m B 1 j x j ∑ j = 1 m B 2 j x j ⋯ ∑ j = 1 m B mj x j
= [ B 11 B 21 ⋯ B m 1 B 12 B 22 ⋯ B m 2 ⋮ ⋮ ⋱ ⋮ B 1 m B 2 m ⋯ B m m ] [ x 1 x 2 ⋮ x m ] + [ B 11 B 12 ⋯ B 1 m B 21 B 22 ⋯ B 2 m ⋮ ⋮ ⋱ ⋮ B m 1 B m 2 ⋯ B m m ] [ x 1 x 2 ⋮ x m ] = \begin{bmatrix} B_{11} & B_{21} & \cdots & B_{m1} \\ B_{12} & B_{22} & \cdots & B_{m2} \\ \vdots & \vdots & \ddots & \vdots \\ B_{1m} & B_{2m} & \cdots & B_{mm} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_m \end{bmatrix}+ \begin{bmatrix} B_{11} & B_{12} & \cdots & B_{1m} \\ B_{21} & B_{22} & \cdots & B_{2m} \\ \vdots & \vdots & \ddots & \vdots \\ B_{m1} & B_{m2} & \cdots & B_{mm} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_m \end{bmatrix} = B 11 B 12 ⋮ B 1 m B 21 B 22 ⋮ B 2 m ⋯ ⋯ ⋱ ⋯ B m 1 B m 2 ⋮ B mm x 1 x 2 ⋮ x m + B 11 B 21 ⋮ B m 1 B 12 B 22 ⋮ B m 2 ⋯ ⋯ ⋱ ⋯ B 1 m B 2 m ⋮ B mm x 1 x 2 ⋮ x m
= ( A T + A ) x = ( A + A T ) x = (A^{T} + A)x = (A + A^{T})x = ( A T + A ) x = ( A + A T ) x
