Definition: a parametric probabilistic model or, in short, parametric model is a family of probability distributions that can be described using a finite number of parameters. The set of all allowable values for the parameter, or the parameter space, is denoted as $\Theta \subseteq \mathbb{R}^k$, and the parametric model is denoted as

$$\mathcal{P} = \{ P_{\boldsymbol \theta}\ |\ \boldsymbol \theta \in \Theta \}$$

Categories of Parametric Models

Exponential Family

Definition: A parametric model is called an exponential family, if it has the form

$$f( \mathbf{x} \ | \ \boldsymbol \theta ) = c(\boldsymbol \theta) h(\mathbf{x}) \exp \left\{ \sum_{i=1}^l w_i(\boldsymbol \theta) t_i(\mathbf{x}) \right\}$$

Here $c(\cdot), h(\cdot)$ are nonnegative, and all functions are real-valued.

In addition,

  1. If k < l, the parametric model is called a curved exponential family.
  2. If k = l, it is called a full exponential family.

Examples of exponential families:

  1. Discrete distributions: Binomial, Poisson, negative binomial;
  2. Continuous distributions: Normal, Gamma, Beta.

Theorem: (Natural Parameterization) For an exponential family, denote natural parameters $\eta_i = w_i(\boldsymbol \theta)$, then the parametric model can be rewritten as

$$f( \mathbf{x} \ | \ \boldsymbol \eta ) = c^{*}(\boldsymbol \eta) h(\mathbf{x}) \exp \left\{ \sum_{i=1}^l \eta_i t_i(\mathbf{x}) \right\}$$

And the natural parameter space defined as $\left\{ \boldsymbol \eta \ \big| \ \int_{\mathbb{R}} h(\mathbf{x}) \exp \left\{ \sum_{i=1}^l \eta_i t_i(\mathbf{x}) \right\} \text{d}x < \infty \right\}$ is convex.

Location-Scale Family

Definition: the location-scale family with standard PDF $f(x)$ is the parametric model

$$\mathcal{P} = \left\{ f_{(\mu,\sigma)} = \frac{1}{\sigma} f(\frac{x-\mu}{\sigma}) \ \bigg| \ (\mu,\sigma) \in \mathbb{R}\times\mathbb{R}^{+} \right\}$$

Here, $\mu$ is called the location parameter, and $\sigma$ is called the scale parameter.


$$X \sim f_{(\mu,\sigma)} \Leftrightarrow \exists Z \sim f(z), \text{ s.t. } X = \sigma Z + \mu$$

🏷 Category=Statistics