In Cooperation with:

American Society for Quality Statistics Division

American Statistical Association

Bernoulli Society for Mathematical Statistics and Probability

Institute of Mathematical Statistics

International Biometric Society

International Chinese Statistical Association

International Society for Bayesian Analysis

International Statistical Institute

Royal Statistical Society

Statistical Society of Canada / Société statistique du Canada

Multivariate Reduced-Rank Regression

Multivariate Reduced-Rank Regression

Alan J. Izenman

Department of Statistics

Temple University

Philadelphia, PA 19122

alan@temple.edu

Key Words: Canonical variate analysis; Dimensionality reduction; Effective dimensionality; Graphical methods; Least-squares estimation; Linear discriminant analysis; Multivariate analysis; Principal component analysis; Regression coefficient matrix.

Multivariate reduced-rank regression is a way of constraining the multivariate linear regression model so that the rank of the regression coefficient matrix has less than full rank. Without the constraint, multivariate linear regression has no true multivariate content.

To see this, suppose we have a random -vector ${\bf X} = (X_1, \cdots, X_r)^\tau$ of predictor (or input) variables with mean vector $\mu$ and covariance matrix ${\bf\Sigma}_{XX}$ , and a random -vector ${\bf Y} = (Y_1, \cdots, Y_s)^\tau$ of response (or output) variables with mean vector $\mu$ and covariance matrix ${\bf\Sigma}_{YY}$ . Suppose that the -vector ${\bf Z} = ({\bf X}^\tau, {\bf Y}^\tau)^\tau$ has a joint distribution with mean vector and covariance matrix,

$\displaystyle \mbox{\boldmath$\mu$}$ $\displaystyle _Z = \left( \begin{array}{c} \mbox{\boldmath$\mu$}_X \\ \mbox{\bo... ... & {\bf\Sigma}_{XY} \\ {\bf\Sigma}_{YX} & {\bf\Sigma}_{YY} \end{array} \right),$

(1)

respectively, where we assume that ${\bf\Sigma}_{XX}$ and ${\bf\Sigma}_{YY}$ are both nonsingular. Now, consider the classical multivariate linear regression model,

$\displaystyle \stackrel{s\times 1}{\bf Y} = \stackrel{s\times 1}{\mbox{\boldmat... ...mes r}{\bf\Theta} \ \stackrel{r\times 1}{\bf X} + \stackrel{s\times 1}{\cal E},$

(2)

where Y depends linearly on X, $\mu$ is the overall mean vector, ${\bf\Theta}$ is the multivariate regression coefficient matrix, and ${\cal E}$ is the error term. In this model, $\mu$ and ${\bf\Theta}$ are unknown and are to be estimated. The least-squares estimator of $\mu$ $, {\bf\Theta})$ is given by

$\displaystyle ($ $\displaystyle \mbox{\boldmath$\mu$}$ $\displaystyle ^*, {\bf\Theta}^*) = {\rm arg} \min_{\mbox{\boldmath$\mu$}, \bf\T... ...\Theta}{\bf X}) ({\bf Y} - \mbox{\boldmath$\mu$} - {\bf\Theta}{\bf X})^\tau \},$

(3)

where

$\displaystyle \mbox{\boldmath$\mu$}$ $\displaystyle ^* =$ $\displaystyle \mbox{\boldmath$\mu$}$ $\displaystyle _Y - {\bf\Theta}^*$ $\displaystyle \mbox{\boldmath$\mu$}$ $\displaystyle _X, \ \ \ {\bf\Theta}^* = {\bf\Sigma}_{YX}{\bf\Sigma}_{XX}^{-1}.$

(4)

In (3), the expectation is taken over the joint distribution of $({\bf X}^\tau, {\bf Y}^\tau)^\tau$ . The minimum achieved is ${\bf\Sigma}_{YY} - {\bf\Sigma}_{YX}{\bf\Sigma}_{XX}^{-1}{\bf\Sigma}_{XY}$ . The $(s\times r)$ -matrix ${\bf\Theta}^*$ is called the (full-rank) regression coefficient matrix. This solution is identical to that obtained by performing a sequence of ordinary least-squares multiple regressions. For the th such multiple regression, is regressed on the -vector X, where $j=1,2,\ldots, s$ . Suppose the minimizing regression coefficient vectors are the -vectors $\beta$ , $j=1,2,\ldots, s$ . Arranging the coefficient vectors as the columns, $\beta$ $_1^{*}, \cdots,$ $\beta$ $_r^{*})$ , of an $(r\times s)$ -matrix, and then transposing the result, it follows from (4) that

$\displaystyle {\bf\Theta}^* = ($ $\displaystyle \mbox{\boldmath$\beta$}$ $\displaystyle _1^{*}, \cdots,$ $\displaystyle \mbox{\boldmath$\beta$}$ $\displaystyle _r^{*})^\tau.$

(5)

Thus, multivariate linear regression is equivalent to just carrying out a sequence of multiple regressions. This is why multivariate regression is often confused with multiple regression.

Now, rewrite the multivariate linear model as

$\displaystyle \stackrel{s\times 1}{\bf Y} = \stackrel{s\times 1}{\mbox{\boldmat... ...s\times r}{\bf C} \ \stackrel{r\times 1}{\bf X} + \stackrel{s\times 1}{\cal E},$

(6)

where the rank constraint is

$\displaystyle {\rm rank}({\bf C}) = t \leq \min(r, s).$

(7)

Equations (6) and (7) form the multivariate reduced-rank regression model. When the rank condition (7) holds, there exist two (nonunique) full-rank matrices A and B, where A is an $(s\times t)$ -matrix and B is a $(t\times r)$ -matrix, such that

$\displaystyle \stackrel{s\times r}{\bf C} = \stackrel{s\times t}{\bf A} \ \stackrel{t\times r}{\bf B}.$

(8)

The multivariate reduced-rank regression model can now be written as

$\displaystyle \stackrel{s\times 1}{\bf Y} = \stackrel{s\times 1}{\mbox{\boldmat... ...t\times r}{\bf B} \ \stackrel{r\times 1}{\bf X} + \stackrel{s\times 1}{\cal E}.$

(9)

The rank condition has been embedded into the regression model. The goal is to estimate $\mu$ , A, and B (and, hence, C).

Let ${\bf\Gamma}$ be a positive-definite symmetric $(s\times s)$ -matrix of weights. The weighted least-squares estimates of $\mu$ $, {\bf A}, {\bf B})$ are

$\displaystyle ($ $\displaystyle \mbox{\boldmath$\mu$}$ $\displaystyle ^*, {\bf A}^*, {\bf B}^*) = {\rm arg} \min_{\mbox{\boldmath$\mu$}... ...$\mu$}-{\bf ABX})^\tau {\bf\Gamma} ({\bf Y}-\mbox{\boldmath$\mu$}-{\bf ABX}) \}$

(10)

where

$\displaystyle \mbox{\boldmath$\mu$}$ $\displaystyle ^*$	$\displaystyle =$	$\displaystyle \mbox{\boldmath$\mu$}$ $\displaystyle _Y - {\bf AB}$ $\displaystyle \mbox{\boldmath$\mu$}$ $\displaystyle _X$	(11)
$\displaystyle {\bf A}^*$	$\displaystyle =$	$\displaystyle {\bf\Gamma}^{-1/2}{\bf V}$	(12)
$\displaystyle {\bf B}^*$	$\displaystyle =$	$\displaystyle {\bf V}^\tau{\bf\Gamma}^{1/2}{\bf\Sigma}_{YX}{\bf\Sigma}_{XX}^{-1},$	(13)

and ${\bf V} = ({\bf v}_1, \cdots, {\bf v}_t)$ is an $(s\times t)$ -matrix, where the th column, ${\bf v}_j$ , is the eigenvector corresponding to the th largest eigenvalue, $\lambda_j$ , of the $(s\times s)$ symmetric matrix,

$\displaystyle {\bf\Gamma}^{1/2} {\bf\Sigma}_{YX} {\bf\Sigma}_{XX}^{-1} {\bf\Sigma}_{XY} {\bf\Gamma}^{1/2}.$

(14)

The multivariate reduced-rank regression coefficient matrix C with rank is, therefore, given by

$\displaystyle {\bf C}^* = {\bf\Gamma}^{-1/2} \left(\sum_{j=1}^t {\bf v}_j{\bf v}_j^\tau\right) {\bf\Gamma}^{1/2}{\bf\Sigma}_{YX}{\bf\Sigma}_{XX}^{-1}.$

(15)

The minimum achieved is ${\rm tr}\{{\bf\Sigma}_{YY}{\bf\Gamma}\} - \sum_{j=1}^t \lambda_j$ .

The main reason that multivariate reduced-rank regression is so important is that it contains as special cases the classical statistical techniques of principal component analysis, canonical variate and correlation analysis, linear discriminant analysis, exploratory factor analysis, multiple correspondence analysis, and other linear methods of analyzing multivariate data. It is also closely related to artificial neural network models and to cointegration in the econometric literature.

For example, the special cases of principal component analysis, canonical variate and correlation analysis, and linear discriminant analysis are given by the following choices: For principal component analysis, set ${\bf X} \equiv {\bf Y}$ and ${\bf\Gamma} = {\bf I}_s$ ; for canonical variate and correlation analysis, set ${\bf\Gamma} = {\bf\Sigma}_{YY}^{-1}$ ; for linear discriminant analysis, use the canonical-variate analysis choice of ${\bf\Gamma}$ and set Y to be a vector of binary variables whose component values (0 or 1) indicate the group or class to which an observation belongs. Details of these and other special cases can be found in Izenman (2008). If the elements of ${\bf\Sigma}_{ZZ}$ in (1) are unknown, as will happen in most practical problems, they have to be estimated using sample data on Z.

The relationships between multivariate reduced-rank regression and the classical linear dimensionality reduction techniques become more interesting when the metaparameter is unknown and has to be estimated. The value of is called the effective dimensionality of the multivariate regression (Izenman, 1980). Estimating is equivalent to the classical problems of determining the number of principal components to retain, the number of canonical variate to retain, or the number of linear discriminant functions necessary for classification purposes. Graphical methods for estimating include the scree plot, the rank trace plot, and heatmap plots. Formal hypothesis tests have also been developed for estimating .

When the number of variables is greater than the number of observations, some adjustments to the results have to be made to ensure that ${\bf\Sigma}_{XX}$ and ${\bf\Sigma}_{YY}$ can be inverted. One simple way of doing this is to replace ${\bf\Sigma}_{XX}$ by ${\bf\Sigma}_{XX} + \delta{\bf I}_r$ and to replace ${\bf\Sigma}_{YY}$ by ${\bf\Sigma}_{YY} + \kappa{\bf I}_s$ as appropriate, where $\delta>0$ and $\kappa>0$ . Other methods, including regularization, banding, tapering, and thresholding, have been studied for estimating large covariance matrices and can be used here as appropriate.

The multivariate reduced-rank regression model can also be developed for the case of nonstochastic (or fixed) predictor variables.

The multivariate reduced-rank regression model has its origins in Anderson (1951), Rao (1964, 1965), and Brillinger (1969), and its name was coined by Izenman (1972, 1975). For the asymptotic distribution of the estimated reduced-rank regression coefficient matrix, see Anderson (1999), who gives results for both the random-X and fixed-X cases. Additional references are the monographs by van der Leeden (1990) and Reinsel and Velu (1998).

Reprinted with permission from Lovric, Miodrag (2011), International Encyclopedia of Statistical Science. Heidelberg: Springer Science +Business Media, LLC

References

Anderson, T.W. (1951). Estimating linear restrictions on regression coefficients for multivariate normal distributions, The Annals of Mathematical Statistics, 22, 327-351.

Anderson, T.W. (1999). Asymptotic distribution of the reduced-rank regression estimator under general conditions, The Annals of Statistics, 27, 1141-1154.

Brillinger, D.R. (1969). The canonical analysis of stationary time series, in Multivariate Analysis II (ed. P.R. Krishaiah), New York: Academic Press, pp. 331-350.

Izenman, A.J. (1972). Reduced-Rank Regression for the Multivariate Linear Model, Its Relationship to Certain Multivariate Techniques, and Its Application to the Analysis of Multivariate Data, Ph.D. dissertation, University of California, Berkeley.

Izenman, A.J. (1975). Reduced-rank regression for the multivariate linear model, Journal of Multivariate Analysis, 5, 248-264.

Izenman, A.J. (1980). Assessing dimensionality in multivariate regression, in Krishnaiah (ed.), Handbook of Statistics, 1, Amsterdam: North-Holland, pp. 571-591.

Izenman, A.J. (2008). Modern Multivariate Statistical Techniques: Regression, Classification, and Manifold Learning, New York: Springer.

Rao, C.R. (1964). The use and interpretation of principal components in applied research, Sankhya (A), 26, 329-358.

Rao, C.R. (1965). Linear Statistical Inference and Its Applications, New York: Wiley.

Reinsel, G.C. and Velu, R.P. (1998). Multivariate Reduced-Rank Regression, Lecture Notes in Statistics, 136, New York: Springer.

Van der Leeden, R. (1990). Reduced-Rank Regression With Structured Residuals, Leiden: DSWO Press.

Multivariate Reduced-Rank Regression is owned by Alan J Izenman.

How to Cite This Entry

Alan J Izenman. "Multivariate Reduced-Rank Regression" (version 1). StatProb: The Encyclopedia Sponsored by Statistics and Probability Societies. Freely available at http://statprob.com/encyclopedia/MultivariateReducedRankRegression.html

Classification

AMS MSC:	62-07 (Statistics :: Data analysis)
	62-09 (Statistics :: Graphical methods)
	62H20 (Statistics :: Multivariate analysis :: Measures of association )
	62H25 (Statistics :: Multivariate analysis :: Factor analysis and principal components; correspondence analysis)
	62H30 (Statistics :: Multivariate analysis :: Classification and discrimination; cluster analysis)
	62J05 (Statistics :: Linear inference, regression :: Linear regression)
	62J07 (Statistics :: Linear inference, regression :: Ridge regression; shrinkage estimators)

Pending Errata and Addenda

None.

Discussion

No messages.