A parametric probabilistic model $\mathcal{P} = \{ P_{\boldsymbol \theta}\mid\boldsymbol \theta \in \Theta \}$ is a family of probability distributions that can be specified using a finite number of parameters $\theta$. The parameter space $\Theta \subseteq \mathbb{R}^k$ is the set of all allowable values for the parameter.

Categories of Parametric Models

Exponential Family

A parametric model is called an exponential family, if it has the form $f( \mathbf{x} \mid \boldsymbol \theta ) = c(\boldsymbol \theta) h(\mathbf{x}) \exp \{ \sum_{i=1}^l w_i(\boldsymbol \theta) t_i(\mathbf{x}) \}$, where $c$ and $h$ are nonnegative, and all functions are real-valued. Additionally, if $k < l$, the parametric model is called a curved exponential family; if $k = l$, it is called a full exponential family.

Examples of exponential families:

  1. Discrete distributions: Binomial, Poisson, negative binomial;
  2. Continuous distributions: Normal, Gamma, Beta.

Theorem (Natural Parameterization): For an exponential family, denote natural parameters $\eta_i = w_i(\boldsymbol \theta)$, then the parametric model can be rewritten as $f(\mathbf{x} \mid \boldsymbol \eta) = c^∗(\boldsymbol \eta) h(\mathbf{x}) \exp \{ \sum_{i=1}^l \eta_i t_i(\mathbf{x}) \}$. And the natural parameter space $\{ \boldsymbol \eta \mid \int f(\mathbf{x} \mid \boldsymbol \eta)~\mathrm{d}\mathbf{x} < \infty \}$ is convex.

Location-Scale Family

The location-scale family with standard PDF $f(x)$ is the parametric model $\mathcal{P} = \{ f_{(\mu,\sigma)} = \frac{1}{\sigma} f(\frac{x-\mu}{\sigma}) \mid (\mu,\sigma) \in \mathbb{R}\times\mathbb{R}^+ \}$ with location parameter $\mu$ and scale parameter $\sigma$.

Theorem: $X \sim f_{(\mu,\sigma)} \Leftrightarrow \exists Z \sim f(z)$, s.t. $X = \sigma Z + \mu$.

Wrapped Distributions

A wrapped probability distribution $p_w(\theta) = \sum_{k \in \mathbb{Z}} p(\theta + 2\pi k)$ on a unit n-sphere is the distribution of random variable $Z = \arg e^{i \phi}$, where $\phi \sim p$ with support $\mathbb{R}$.

Parametric Families by Support

  1. Discrete:
    1. finite: Bernoulli, binomial, categorical, multinomial, hypergeometric, Zipf;
    2. semi-infinite: Poisson, geometric, negative binomial;
  2. Continuous (Euclidean):
    1. bounded: uniform, triangular, semicircle, Beta, logit-normal;
    2. semi-infinite: exponential, Erlang, Gamma, inverse Gamma, generalized inverse Gaussian; Pareto, Fréchet, Weibull; Chi-squared, Chi, F; log-normal, log-Cauchy, log-Laplace, log-logistic;
    3. infinite: Gaussian, Cauchy, Student's t, Fisher's z; Laplace, logistic, Gumbel;
  3. Manifold:
    • simplex: Dirichlet;
    • circle: von Mises;
    • sphere: von Mises–Fisher [@Fisher1953], Bingham [@Bingham1974], Kent [@Kent1982];
    • torus: bivariate von Mises [@Mardia1975];
    • Stiefel and Grassmann manifolds: matrix Langevin @Chikuse2003;

Stiefel manifold $V_{k, m}$ is the space of k-frames in the Euclidean m-space: $X \in M_{m, k}, X' X = I$. Grassmann manifold $G_{k, m-k}$ is the space of rank-k projection matrices, or equivalently, k-subspaces in the Euclidean m-space. When $k = 1$, the Stiefel manifold coincides with the $(m-1)$-sphere, and the Grassmann manifold coincides with the manifold of undirected axes/lines.


🏷 Category=Statistics