For a matrix $M \in M_{m,n}$, a pseudoinverse or generalized inverse is a matrix $M^+ \in M_{n,m}$ that has some properties analogous to the inverse of an invertible matrix. Left inverse of an injective matrix is a generalized inverse whose composition with the matrix is the identity map on the domain: $\text{rank}(M) = n$, $\exists M^+ \in M_{n,m}$: $M^+ M = I_n$. Right inverse of a surjective matrix is a generalized inverse such that the composition of the matrix with it is the identity map on the codomain: $\text{rank}(M) = m$, $\exists M^+ \in M_{n,m}$: $M M^+ = I_m$. Left and right inverses are symmetric concepts: if M is a left inverse of N, then N is a right inverse of M. Left inverse and right inverse are not unique: the set of left inverses of a matrix $M \in M^∗_{m,n}$, m < n, is $[M^+] = \{(N^T M)^{-1} N^T : N \in M^∗_{m,n}, N^T M \in \text{GL}_n\}$; notice that the element is invariant under $N \to N A$ where $A \in \text{GL}_n$; $[M^+]$ is bijective to the Grassmann manifold $G_{n,m}$ minus the measure-zero set of subspaces with vectors orthogonal to M.

Moore-Penrose inverse

Moore-Penrose inverse of an m-by-n matrix [@Moore1920; @Bjerhammar1951; @Penrose1955] is an n-by-m matrix that satisifes the following four Penrose equations [@Ben-Israel2003, Sec 1.1]: (1) $M M^\dagger M = M$; (2) $M^\dagger M M^\dagger = M^\dagger$; (3) $(M M^\dagger)^∗ = M M^\dagger$; and (4) $(M^\dagger M)^∗ = M^\dagger M$. Or equivalently, one that satisifes the following two equations: $M M^\dagger = P_{[M]}$ and $M^\dagger M = P_{[M^∗]}$, where $P_{[M]}$ denotes the orthogonal projector onto the span of M.

Properties:

  • $M^\dagger$ exists and is unique; or equivalently, $\dagger: M_{m,n} \cong M_{n,m}$.
  • Involution: $(M^\dagger)^\dagger = M$.
  • Fixed points: $M^\dagger = M$ if and only if M is an orthogonal projector; or equivalently, $\dagger \cap \text{Id} = \mathcal{P}(k,n)$.
  • Representation: let $M = V \Sigma W^∗$ be a compact SVD, then $M^\dagger = W \Sigma^{-1} V^∗$.

Relation with dual operator (Hermitian adjoint matrix):

  • $M^\dagger$ equals $M^∗$ with nonzero singular values replaced by their reciprocals: $M^\dagger = W \Sigma^{-1} V^∗$, $M^∗ = W \Sigma V^∗$.
  • $M^\dagger$ and $M^∗$ have the same fundamental subspaces: $\text{ker}(M^\dagger) = \text{ker}(M^∗)$, $\text{im}(M^\dagger) = \text{im}(M^∗)$.

Relation with matrix inverse:

  • If M is invertible, then $M^\dagger = M^{-1}$; $\dagger|_{\text{GL}_n} = (\cdot)^{-1}$.
  • If M has full column rank, then $M^\dagger = (M^∗ M)^{-1} M^∗$, which is a left inverse: $M^\dagger M = I$.
  • If M has full row rank, then $M^\dagger = M^∗ (M M^∗)^{-1}$, which is a right inverse: $M M^\dagger = I$.

Relation with linear equations / model fitting:

  • $X^\dagger y$ is the minimum Euclidean norm solution to $X \beta = y$, if $y \in \text{im}(X)$.
  • $X^\dagger y$ is the least squares solution to $X \beta = y$, if $y \notin \text{im}(X)$.

Operation equalities (see [@Peterson-Minka] for more):

  • It commutes with transposition and conjugation: $(M^T)^\dagger = (M^\dagger)^T$, $\overline{M}^\dagger = \overline{M^\dagger}$.
  • $(M N)^\dagger = (M^\dagger M N)^\dagger (M N N^\dagger)^\dagger$.
  • Cases when $(M N)^\dagger = N^\dagger M^\dagger$: (1) M and N have full rank; (2) $N = M^∗$; or (3) $N = M^T$.
  • A, B, C have full rank $\not\Rightarrow (A B C)^\dagger = C^\dagger B^\dagger A^\dagger$. For example, let $c = (c_1, c_2)$, then $(e_1 e_2^T c)^\dagger = c_2^\dagger e_1^T$ and $c^\dagger (e_2^T)^\dagger e_1^\dagger = (c_1^2+c_2^2)^\dagger c_2 e_1^T$, which are not equal as long as $c_1, c_2 \ne 0$.
  • $M^\dagger = (M^T M)^\dagger M^T$, $M^\dagger = M^T (M M^T)^\dagger$; (can replace transpose with Hermitian conjugate).
  • Let $P \in \mathcal{P}(k,m)$, then $(P M)^\dagger P = (P M)^\dagger$; let $P \in \mathcal{P}(k,n)$, then $P (M P)^\dagger = (M P)^\dagger$.

🏷 Category=Algebra Category=Matrix Analysis