Transformation

Univariate transformation

Theorem (Monotone transformation): Let $X \sim F_X(x)$, $Y=g(X)$:

  1. If $g(x)$ is increasing on $\text{supp}(X)$, then $Y \sim F_X(g^{-1}(y))$ for $y \in g(\text{supp}(X))$;
  2. If $g(x)$ is decreasing on $\text{supp}(X)$ and $F_X(x)$ is continuous, then $Y \sim 1 - F_X(g^{-1}(y))$ for $y \in g(\text{supp}(X))$;

Corollary (Probability integral transformation):

  1. If $X \sim F_X$, $F_X$ is continuous, then $F_X(X) \sim U(0,1)$;
  2. If $U \sim U(0,1)$, $F$ is a continuous CDF, then $F^{-1}(U) \sim F$;
  3. If $U \sim U(0,1)$, $\bar{F}$ is a continuous tail distribution, then $\bar{F}^{-1}(U) \sim F$;

Thus, any random variable with continuous distribution $F(x)$ can be simulated as either $F^{-1}(u)$ or $\bar{F}^{-1}(u)$, where $u$ is sampled from the standard uniform distribution.

Multivariate transformation

For $Z = g(X,Y)$, the distribution can be found by either of the following:

  1. Direct method: Let $G_z = \{ (x,y) \,|\, Z \leq z \}$, then $F_Z (z) = \iint_{G_z} f_{X,Y}(x,y) \,\text{d}x \text{d}y$.
  2. Conditional method: Find the conditional distribution of univariate transformation $Z|X = g(X,Y|X)$, then $f_Z(z) = \mathbb{E} f_{Z|X}(z|X)$.

These methods can be applied to general multivariate transformations.

Theorem (Summation of independent random variables): If $X ∐ Y$, $Z = X+Y$, then $f_Z = f_X * f_Y$ and $\Phi_Z = \Phi_X \cdot \Phi_Y$.

Coordinate transformation

Let $\mathbf{g}: \mathbb{R}^n \to \mathbb{R}^n$ with nonzero Jacobian determinant $J_{\mathbf{g}} (\mathbf{x})$, $\mathbf{Y} = \mathbf{g}(\mathbf{X})$, then $f_{\mathbf{Y}}(\mathbf{y}) = \sum_{ \{ \mathbf{x} | \mathbf{g}(\mathbf{x}) = \mathbf{y} \} } \dfrac{ f_{\mathbf{X}}(\mathbf{x}) }{ \lvert J_{\mathbf{g}} (\mathbf{x}) \rvert }$.

Special cases:

  • Linear/Affine transformation: Let $\mathbf{Y} = A \mathbf{X} + \mathbf{b}$, then $f_{\mathbf{Y}} (\mathbf{y}) = \dfrac{ f_{\mathbf{X}}( A^{-1} (\mathbf{y} - \mathbf{b}) ) } { \lvert \det(A) \rvert }$ and $\Phi_{\mathbf{Y}} (\mathbf{v}) = e^{i \mathbf{v} \cdot \mathbf{b}} \Phi_X(A^T \mathbf{v})$.
  • Polar coordinate transformation: Let $(X,Y) \to (R, \Theta)$, then $f_{R,\Theta} (r,\theta) = r f_{X,Y} (r \cos\theta, r \sin\theta)$.

Decomposition

A probability distribution is infinitely divisible if it can be expressed as the probability distribution of the sum of an arbitrary number of independent and identically distributed random variables.

Gaussian distribution is infinitely divisible.

Cramer's decomposition theorem: If the sum of two independent real-valued random variables is Gaussian, then both of them must also be Gaussian.

The result can be generated to any finite sum by induction. Proof of this theorem seems nontrivial.

Examples with Common Probabilistic Models

All random numbers start from uniform: let $U \sim U(0, 1)$,

  • $\dfrac{- \ln U}{\lambda} \sim \text{Exp}(\lambda)$;
  • $\dfrac{1}{U} \sim \text{Pareto}(1, 1)$;
  • $\Phi^{-1}(U) \sim N(0, 1)$;

Uniform Order Statistics:

  • For a random sample of size $n$, $U_{(r)} \sim B(r,n+1-r)$;
  • $a, b \in \mathbb{N}$, $a < b \le n$, then $U_{(b)} - U_{(a)} \sim U_{(b-a)}$;
  • $X_i \text{ i.i.d. } \text{Exp}(1)$, $S_k = \sum_{i=1}^k X_i$, then $\dfrac{(S_1, \cdots, S_n)}{S_{n+1}} \sim ( U_{(1)}, \cdots, U_{(n)} )$;

Beta:

  • $X \sim B(\alpha,\beta), Y \sim B(\alpha+\beta, \gamma)$, $X ∐ Y$, then $XY \sim B(\alpha, \beta + \gamma)$;

Exponential:

  • $X \sim \text{Exp}(1)$, then $X^{1/k} \sim \text{Weibull}(1, k)$;
  • $X \sim \text{Exp}(1)$, then $\dfrac{X^{-\xi}-1}{\xi} \sim \text{GEV}(0, 1, \xi)$;
  • $X \sim \text{Exp}(1)$, then $\dfrac{-1}{X} \sim e^{-1/x} x^{-2}$, a regularly varying (Pareto-like) distribution;
  • $X_i \text{ i.i.d. } \text{Exp}(\lambda)$, then $\sum_{i=1}^k X_i \sim \text{Erlang}(k, \lambda)$;
  • $X,Y \sim \text{Exp}(\lambda)$, $X ∐ Y$, then $X - Y \sim \text{Laplace}(0, \frac{1}{\lambda})$;
  • $X \sim \text{Exp}(\lambda_x)$, $Y \sim \text{Exp}(\lambda_y)$, $X ∐ Y$, then $X+Y \sim \dfrac{\lambda_x P_Y - \lambda_y P_X}{\lambda_x - \lambda_y}$, ($\lambda_x \ne \lambda_y$);

Gamma/Erlang:

  • $X \sim \Gamma(\alpha, 1)$, then $X / \beta \sim \Gamma(\alpha, \beta)$;
  • $X_1 \sim \Gamma(\alpha_1,\beta)$, $X_2 \sim \Gamma(\alpha_2,\beta)$, $X_1 ∐ X_2$, then $X_1 + X_2 \sim \Gamma(\alpha_1+\alpha_2,\beta)$;

Poisson:

  • $X \sim \text{Poisson}(\lambda_x)$, $Y \sim \text{Poisson}(\lambda_y)$, $X ∐ Y$, then $X + Y \sim \text{Poisson}(\lambda_x + \lambda_y)$;

Gaussian and friends:

  • $X \sim N(\mu, \sigma^2)$, then $e^X \sim \text{LnN}(\mu, \sigma^2)$;
  • $X, Y \sim N(0, 1)$, $X ∐ Y$, then $\dfrac{X}{Y} \sim \text{Cauchy}(1)$;
  • $X_i \text{ i.i.d. } N(0,1)$, then $\sum_{i=1}^n X_i^2 \sim \chi^2(n) \sim \Gamma \left(\frac{n}{2}, \frac{1}{2} \right)$;

Click here for all the proofs.

🏷 Category=Probability