A parametric probabilistic model $\mathcal{P} = \{ P_{\boldsymbol \theta}\mid\boldsymbol \theta \in \Theta \}$ is a family of probability distributions that can be specified using a finite number of parameters $\theta$. The parameter space $\Theta \subseteq \mathbb{R}^k$ is the set of all allowable values for the parameter.

Categories of Parametric Models

Exponential Family

A parametric model is called an exponential family, if it has the form $f( \mathbf{x} \mid \boldsymbol \theta ) = c(\boldsymbol \theta) h(\mathbf{x}) \exp \{ \sum_{i=1}^l w_i(\boldsymbol \theta) t_i(\mathbf{x}) \}$, where $c$ and $h$ are nonnegative, and all functions are real-valued. Additionally, if $k < l$, the parametric model is called a curved exponential family; if $k = l$, it is called a full exponential family.

Examples of exponential families:

  1. Discrete distributions: Binomial, Poisson, negative binomial;
  2. Continuous distributions: Normal, Gamma, Beta.

Theorem (Natural Parameterization): For an exponential family, denote natural parameters $\eta_i = w_i(\boldsymbol \theta)$, then the parametric model can be rewritten as $f(\mathbf{x} \mid \boldsymbol \eta) = c^∗(\boldsymbol \eta) h(\mathbf{x}) \exp \{ \sum_{i=1}^l \eta_i t_i(\mathbf{x}) \}$. And the natural parameter space $\{ \boldsymbol \eta \mid \int f(\mathbf{x} \mid \boldsymbol \eta)~\mathrm{d}\mathbf{x} < \infty \}$ is convex.

Location-Scale Family

The location-scale family with standard PDF $f(x)$ is the parametric model $\mathcal{P} = \{ f_{(\mu,\sigma)} = \frac{1}{\sigma} f(\frac{x-\mu}{\sigma}) \mid (\mu,\sigma) \in \mathbb{R}\times\mathbb{R}^+ \}$ with location parameter $\mu$ and scale parameter $\sigma$.

Theorem: $X \sim f_{(\mu,\sigma)} \Leftrightarrow \exists Z \sim f(z)$, s.t. $X = \sigma Z + \mu$.

Wrapped Distributions

A wrapped probability distribution $p_w(\theta) = \sum_{k \in \mathbb{Z}} p(\theta + 2\pi k)$ on a unit $n$-sphere is the distribution of random variable $Z = \arg e^{i \phi}$, where $\phi \sim p$ with support $\mathbb{R}$.

Parametric Families by Support

  1. Discrete:
    1. finite: Bernoulli, binomial, categorical, multinomial, hypergeometric, Zipf;
    2. semi-infinite: Poisson, geometric, negative binomial;
  2. Continuous (Euclidean):
    1. bounded: uniform, triangular, semicircle, Beta, logit-normal;
    2. semi-infinite: exponential, Erlang, Gamma, inverse Gamma, generalized inverse Gaussian; Pareto, Fréchet, Weibull; Chi-squared, Chi, F; log-normal, log-Cauchy, log-Laplace, log-logistic;
    3. infinite: Gaussian, Cauchy, Student's t, Fisher's z; Laplace, logistic, Gumbel;
  3. Manifold:
    • Dirichlet (simplex);
    • von Mises (circle); von Mises–Fisher [@Fisher1953], Bingham [@Bingham1974], Kent @Kent1982; bivariate von Mises @Mardia1975;
    • matrix von Mises-Fisher (aka matrix Langevin) (Stiefel manifold); matrix Bingham (Grassmann manifold);

🏷 Category=Statistics