Total least squares

Short description: Statistical technique

The bivariate (Deming regression) case of total least squares. The red lines show the error in both x and y. This is different from the traditional least squares method which measures error parallel to the y axis. The case shown, with deviations measured perpendicularly, arises when errors in x and y have equal variances. Part of a series on

Regression analysis
Models

* Linear regression
* Simple regression
* Polynomial regression
* General linear model

* Generalized linear model
* Discrete choice
* Binomial regression
* Binary regression
* Logistic regression
* Multinomial logit
* Mixed logit
* Probit
* Multinomial probit
* Ordered logit
* Ordered probit
* Poisson

* Multilevel model
* Fixed effects
* Random effects
* Linear mixed-effects model
* Nonlinear mixed-effects model

* Nonlinear regression
* Nonparametric
* Semiparametric
* Robust
* Quantile
* Isotonic
* Principal components
* Least angle
* Local
* Segmented

* Errors-in-variables

Estimation

* Least squares
* Linear
* Non-linear

* Ordinary
* Weighted
* Generalized

* Partial
* Total
* Non-negative
* Ridge regression
* Regularized

* Least absolute deviations
* Iteratively reweighted
* Bayesian
* Bayesian multivariate

Background

* Regression validation
* Mean and predicted response
* Errors and residuals
* Goodness of fit
* Studentized residual
* Gauss–Markov theorem

* v
* t
* e

In applied statistics, total least squares is a type of errors-in-variables regression, a least squares data modeling technique in which observational errors on both dependent and independent variables are taken into account. It is a generalization of Deming regression and also of orthogonal regression, and can be applied to both linear and non-linear models.

The total least squares approximation of the data is generically equivalent to the best, in the Frobenius norm, low-rank approximation of the data matrix.[1]

## Contents

* 1 Linear model
* 1.1 Background
* 1.2 Allowing observation errors in all variables
* 1.3 Example
* 1.4 Algebraic point of view
* 1.5 Computation
* 2 Non-linear model
* 3 Geometrical interpretation
* 4 Scale invariant methods
* 5 See also
* 6 Notes
* 7 References
* 7.1 Others

## Linear model

### Background

In the least squares method of data modeling, the objective function to be minimized, S, is a quadratic form:

S=𝐫𝐓𝐖𝐫,

where r is the vector of residuals and W is a weighting matrix. In linear least squares the model contains equations which are linear in the parameters appearing in the parameter vector β, so the residuals are given by

𝐫=𝐲−𝐗β.

There are m observations in y and n parameters in β with m>n. X is a m×n matrix whose elements are either constants or functions of the independent variables, x. The weight matrix W is, ideally, the inverse of the variance-covariance matrix 𝐌y of the observations y. The independent variables are assumed to be error-free. The parameter estimates are found by setting the gradient equations to zero, which results in the normal equations[lower-alpha 1]

𝐗𝐓𝐖𝐗β=𝐗𝐓𝐖𝐲.

### Allowing observation errors in all variables

Now, suppose that both x and y are observed subject to error, with variance-covariance matrices 𝐌x and 𝐌y respectively. In this case the objective function can be written as

S=𝐫𝐱𝐓𝐌𝐱−𝟏r𝐱+𝐫𝐲𝐓𝐌𝐲−𝟏r𝐲,

where 𝐫x and 𝐫y are the residuals in x and y respectively. Clearly these residuals cannot be independent of each other, but they must be constrained by some kind of relationship. Writing the model function as 𝐟(r𝐱,r𝐲,β), the constraints are expressed by m condition equations.[2]

𝐅=Δ𝐲−∂f∂rxr𝐱−∂f∂ryr𝐲−𝐗Δβ=𝟎.

Thus, the problem is to minimize the objective function subject to the m constraints. It is solved by the use of Lagrange multipliers. After some algebraic manipulations,[3] the result is obtained.

𝐗𝐓𝐌−𝟏𝐗Δβ=𝐗𝐓𝐌−𝟏Δ𝐲,

or alternatively 𝐗𝐓𝐌−𝟏𝐗β=𝐗𝐓𝐌−𝟏𝐲, where M is the variance-covariance matrix relative to both independent and dependent variables.

𝐌=K𝐱M𝐱𝐊𝐱𝐓+K𝐲M𝐲𝐊𝐲𝐓; K𝐱=−∂f∂rx, K𝐲=−∂f∂ry.

### Example

When the data errors are uncorrelated, all matrices M and W are diagonal. Then, take the example of straight line fitting.

f(xi,β)=α+βxi

in this case

Mii=σy,i2+β2σx,i2

showing how the variance at the ith point is determined by the variances of both independent and dependent variables and by the model being used to fit the data. The expression may be generalized by noting that the parameter β is the slope of the line.

Mii=σy,i2+(dydx)i2σx,i2

An expression of this type is used in fitting pH titration data where a small error on x translates to a large error on y when the slope is large.

### Algebraic point of view

As was shown in 1980 by Golub and Van Loan, the TLS problem does not have a solution in general.[4] The following considers the simple case where a unique solution exists without making any particular assumptions.

The computation of the TLS using singular value decomposition (SVD) is described in standard texts.[5] We can solve the equation

XB≈Y

for B where X is m-by-n and Y is m-by-k. [lower-alpha 2]

That is, we seek to find B that minimizes error matrices E and F for X and Y respectively. That is,

argminB,E,F‖[EF]‖F,(X+E)B=Y+F

where [EF] is the augmented matrix with E and F side by side and ‖⋅‖F is the Frobenius norm, the square root of the sum of the squares of all entries in a matrix and so equivalently the square root of the sum of squares of the lengths of the rows or columns of the matrix.

This can be rewritten as

[(X+E)(Y+F)][B−Ik]=0.

where Ik is the k×k identity matrix. The goal is then to find [EF] that reduces the rank of [XY] by k. Define [U][Σ][V]* to be the singular value decomposition of the augmented matrix [XY].

[XY]=[UXUY][ΣX00ΣY][VXXVXYVYXVYY]*=[UXUY][ΣX00ΣY][VXX*VYX*VXY*VYY*]

where V is partitioned into blocks corresponding to the shape of X and Y.

Using the Eckart–Young theorem, the approximation minimising the norm of the error is such that matrices U and V are unchanged, while the smallest k singular values are replaced with zeroes. That is, we want

[(X+E)(Y+F)]=[UXUY][ΣX000k×k][VXXVXYVYXVYY]*

so by linearity,

[EF]=−[UXUY][0n×n00ΣY][VXXVXYVYXVYY]*.

We can then remove blocks from the U and Σ matrices, simplifying to

[EF]=−UYΣY[VXYVYY]*=−[XY][VXYVYY][VXYVYY]*.

This provides E and F so that

[(X+E)(Y+F)][VXYVYY]=0.

Now if VYY is nonsingular, which is not always the case (note that the behavior of TLS when VYY is singular is not well understood yet), we can then right multiply both sides by −VYY−1 to bring the bottom block of the right matrix to the negative identity, giving[6]

[(X+E)(Y+F)][−VXYVYY−1−VYYVYY−1]=[(X+E)(Y+F)][B−Ik]=0,

and so

B=−VXYVYY−1.

A naive GNU Octave/MATLAB implementation of this is:

function B = tls(X, Y)
[m n]   = size(X);             % n is the width of X (X is m by n)
Z       = [X Y];               % Z is X augmented with Y.
[U S V] = svd(Z, 0);           % find the SVD of Z.
VXY     = V(1:n, 1+n:end);     % Take the block of V consisting of the first n rows and the n+1 to last column
VYY     = V(1+n:end, 1+n:end); % Take the bottom-right block of V.
B       = -VXY / VYY;
end

The way described above of solving the problem, which requires that the matrix VYY is nonsingular, can be slightly extended by the so-called classical TLS algorithm.[7]

### Computation

The standard implementation of classical TLS algorithm written by Van Huffel is available through Netlib, see also.[8][9] All modern implementations based, for example, on solving a sequence of ordinary least squares problems, approximate the matrix B (denoted X in the literature),[lower-alpha 2] as introduced by Van Huffel and Vandewalle. It is worth noting, that this solution is, however, only the TLS solution when rank⁡(W(q,e))=e and rank⁡(V12(−e))=d−e. In other cases it represents a BT-TLS, a solution of the truncated total least squares problem (T-TLS).[10][11] The T-TLS has a regularization property that real TLS does not have.[11]

## Non-linear model

For non-linear systems similar reasoning shows that the normal equations for an iteration cycle can be written as

𝐉𝐓𝐌−𝟏𝐉Δβ=𝐉𝐓𝐌−𝟏Δ𝐲,

where 𝐉 is the Jacobian matrix.

## Geometrical interpretation

When the independent variable is error-free a residual represents the "vertical" distance between the observed data point and the fitted curve (or surface). In total least squares a residual represents the distance between a data point and the fitted curve measured along some direction. In fact, if both variables are measured in the same units and the errors on both variables are the same, then the residual represents the shortest distance between the data point and the fitted curve, that is, the residual vector is perpendicular to the tangent of the curve. For this reason, this type of regression is sometimes called two dimensional Euclidean regression (Stein, 1983)[12] or orthogonal regression.

## Scale invariant methods

A serious difficulty arises if the variables are not measured in the same units. First consider measuring distance between a data point and the line: what are the measurement units for this distance? If we consider measuring distance based on Pythagoras' Theorem then it is clear that we shall be adding quantities measured in different units, which is meaningless. Secondly, if we rescale one of the variables e.g., measure in grams rather than kilograms, then we shall end up with different results (a different line). To avoid these problems it is sometimes suggested that we convert to dimensionless variables—this may be called normalization or standardization. However, there are various ways of doing this, and these lead to fitted models which are not equivalent to each other. One approach is to normalize by known (or estimated) measurement precision thereby minimizing the Mahalanobis distance from the points to the line, providing a maximum-likelihood solution;[13] the unknown precisions could be found via analysis of variance.

In short, total least squares does not have the property of units-invariance--i.e. it is not scale invariant. For a meaningful model we require this property to hold. A way forward is to realise that residuals (distances) measured in different units can be combined if multiplication is used instead of addition. Consider fitting a line: for each data point the product of the vertical and horizontal residuals equals twice the area of the triangle formed by the residual lines and the fitted line. We choose the line which minimizes the sum of these areas. Nobel laureate Paul Samuelson proved in 1942 that, in two dimensions, it is the only line expressible solely in terms of the ratios of standard deviations and the correlation coefficient which (1) fits the correct equation when the observations fall on a straight line, (2) exhibits scale invariance, and (3) exhibits invariance under interchange of variables.[13] This solution has been rediscovered in different disciplines and is variously known as standardised major axis (Ricker 1975, Warton et al., 2006),[14][15] the reduced major axis, the geometric mean functional relationship (Draper and Smith, 1998),[16][lower-alpha 3] least products regression, diagonal regression, line of organic correlation, and the least areas line (Tofallis, 2002).[17] GMFR has been recommended for use in biology.[18]

Tofallis (2015, 2023)[19][20] has extended this approach to deal with multiple variables. The calculations are simpler than for total least squares as they only require knowledge of the covariance matrix, and can be computed using standard spreadsheet functions. The scale-invariant TLS is an example of a symmetric regression, where every variable is treated equally.[20] More concretely, given a covariance matrix M and its inversion M-1, the regression is M11−1x1+M22−1x2+...=constant. For a large number of variables, it is more efficient to directly compute the diagonal elements of the M-1 using the positive definiteness of M.[20]

## See also

* Regression dilution
* Deming regression, a special case with two predictors and independent errors.
* Errors-in-variables model
* Gauss-Helmert model
* Linear regression
* Least squares
* Principal component analysis
* Principal component regression

## Notes

1. ↑ An alternative form is 𝐗𝐓𝐖𝐗Δβ=𝐗𝐓𝐖Δ𝐲, where Δβ is the parameter shift from some starting estimate of β and Δ𝐲 is the difference between y and the value calculated using the starting value of β
2. ↑ 2.0 2.1 The notation XB ≈ Y is used here to reflect the notation used in the earlier part of the article. In the computational literature the problem has been more commonly presented as AX ≈ B, i.e. with the letter X used for the n-by-k matrix of unknown regression coefficients.
3. ↑ So-named because the GMFR slope is the geometric mean of the two OLS slopes, which also makes it easy to calculate using existing tools.

## References

1. ↑ I. Markovsky and S. Van Huffel, Overview of total least squares methods. Signal Processing, vol. 87, pp. 2283–2302, 2007. preprint
2. ↑ W.E. Deming, Statistical Adjustment of Data, Wiley, 1943
3. ↑ Gans, Peter (1992). Data Fitting in the Chemical Sciences. Wiley. ISBN 9780471934127. http://www.wiley.com/WileyCDA/WileyTitle/productCd-0471934127.html. Retrieved 4 December 2012.
4. ↑ G. H. Golub and C. F. Van Loan, An analysis of the total least squares problem. Numer. Anal., 17, 1980, pp. 883–893.
5. ↑ Matrix Computations (3rd ed.). The Johns Hopkins University Press. 1996. pp 596.
6. ↑ Bjõrck, Ake (1996) Numerical Methods for Least Squares Problems, Society for Industrial and Applied Mathematics. ISBN 978-0898713602
7. ↑ S. Van Huffel and J. Vandewalle (1991) The Total Least Squares Problems: Computational Aspects and Analysis. SIAM Publications, Philadelphia PA.
8. ↑ S. Van Huffel, Documented Fortran 77 programs of the extended classical total least squares algorithm, the partial singular value decomposition algorithm and the partial total least squares algorithm, Internal Report ESAT-KUL 88/1, ESAT Lab., Dept. of Electrical Engineering, Katholieke Universiteit Leuven, 1988.
9. ↑ S. Van Huffel, The extended classical total least squares algorithm, J. Comput. Appl. Math., 25, pp. 111–119, 1989.
10. ↑ M. Plešinger, The Total Least Squares Problem and Reduction of Data in AX ≈ B. Doctoral Thesis, TU of Liberec and Institute of Computer Science, AS CR Prague, 2008. Ph.D. Thesis
11. ↑ 11.0 11.1 Hnětynková, Iveta; Plešinger, Martin; Sima, Diana Maria; Strakoš, Zdeněk; Van Huffel, Sabine (July 2011). "The Total Least Squares Problem in AX≈B: A New Classification with the Relationship to the Classical Works". SIAM Journal on Matrix Analysis and Applications 32 (3): 748–770. doi:10.1137/100813348. https://www.sam.math.ethz.ch/sam_reports/reports_final/reports2010/2010-38.pdf.
12. ↑ Stein, Yaakov J.. Two Dimensional Euclidean Regression. http://www.dspcsp.com/pubs/euclreg.pdf.
13. ↑ 13.0 13.1 Samuelson, Paul A. (1942). "A Note on Alternative Regressions". Econometrica 10 (1): 80–83. doi:10.2307/1907024.
14. ↑ Ricker, W. E. (1975). "A note concerning Professor Jolicoeur's Comments". Journal of the Fisheries Research Board of Canada 32 (8): 1494–1498. doi:10.1139/f75-172.
15. ↑ Warton, David I.; Wright, Ian J.; Falster, Daniel S.; Westoby, Mark (2006). "Bivariate line-fitting methods for allometry". Biological Reviews 81 (2): 259–291. doi:10.1017/S1464793106007007. PMID 16573844.
16. ↑ Draper, NR and Smith, H. Applied Regression Analysis, 3rd edition, pp. 92–96. 1998
17. ↑ Tofallis, Chris (2002). "Model Fitting for Multiple Variables by Minimising the Geometric Mean Deviation". in Van Huffel, Sabine; Lemmerling, P.. Total Least Squares and Errors-in-Variables Modeling: Analysis, Algorithms and Applications. Dordrecht: Kluwer Academic Publ.. doi:10.1007/978-94-017-3552-0. ISBN 978-1402004766.
18. ↑ Xu, Shaoji (2014-10-02). "A Property of Geometric Mean Regression". The American Statistician 68 (4): 277–281. doi:10.1080/00031305.2014.962763. ISSN 0003-1305. http://dx.doi.org/10.1080/00031305.2014.962763.
19. ↑ Tofallis, Chris (2015). "Fitting Equations to Data with the Perfect Correlation Relationship". SSRN 2707593.
20. ↑ 20.0 20.1 20.2 Tofallis, Chris (18 September 2023). "Fitting an Equation to Data Impartially". Mathematics 11 (18): 3957. doi:10.3390/math11183957.

### Others

* I. Hnětynková, M. Plešinger, D. M. Sima, Z. Strakoš, and S. Van Huffel, The total least squares problem in AX ≈ B. A new classification with the relationship to the classical works. SIMAX vol. 32 issue 3 (2011), pp. 748–770. Available as a preprint.
* M. Plešinger, The Total Least Squares Problem and Reduction of Data in AX ≈ B. Doctoral Thesis, TU of Liberec and Institute of Computer Science, AS CR Prague, 2008. Ph.D. Thesis
* C. C. Paige, Z. Strakoš, Core problems in linear algebraic systems. SIAM J. Matrix Anal. Appl. 27, 2006, pp. 861–875. doi:10.1137/040616991
* S. Van Huffel and P. Lemmerling, Total Least Squares and Errors-in-Variables Modeling: Analysis, Algorithms and Applications. Dordrecht, The Netherlands: Kluwer Academic Publishers, 2002.
* S. Jo and S. W. Kim, Consistent normalized least mean square filtering with noisy data matrix. IEEE Trans. Signal Process., vol. 53, no. 6, pp. 2112–2123, Jun. 2005.
* R. D. DeGroat and E. M. Dowling, The data least squares problem and channel equalization. IEEE Trans. Signal Process., vol. 41, no. 1, pp. 407–411, Jan. 1993.
* S. Van Huffel and J. Vandewalle, The Total Least Squares Problems: Computational Aspects and Analysis. SIAM Publications, Philadelphia PA, 1991. doi:10.1137/1.9781611971002
* T. Abatzoglou and J. Mendel, Constrained total least squares, in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP’87), Apr. 1987, vol. 12, pp. 1485–1488.
* P. de Groen An introduction to total least squares, in Nieuw Archief voor Wiskunde, Vierde serie, deel 14, 1996, pp. 237–253 arxiv.org.
* G. H. Golub and C. F. Van Loan, An analysis of the total least squares problem. SIAM J. on Numer. Anal., 17, 1980, pp. 883–893. doi:10.1137/0717073
* Perpendicular Regression Of A Line at MathPages
* A. R. Amiri-Simkooei and S. Jazaeri Weighted total least squares formulated by standard least squares theory, in Journal of Geodetic Science, 2 (2): 113–124, 2012 [1].

* v
* t
* e

Least squares and regression analysis

Computational statistics|

* Least squares
* Linear least squares
* Non-linear least squares
* Iteratively reweighted least squares

Correlation and dependence|

* Pearson product-moment correlation
* Rank correlation (Spearman's rho
* Kendall's tau)
* Partial correlation
* Confounding variable

Regression analysis|

* Ordinary least squares
* Partial least squares
* Total least squares
* Ridge regression

Regression as a
statistical model|

| Linear regression|

* Simple linear regression
* Ordinary least squares
* Generalized least squares
* Weighted least squares
* General linear model

|

Predictor structure|

* Polynomial regression
* Growth curve (statistics)
* Segmented regression
* Local regression

Non-standard|

* Nonlinear regression
* Nonparametric
* Semiparametric
* Robust
* Quantile
* Isotonic

Non-normal errors|

* Generalized linear model
* Binomial
* Poisson
* Logistic

Decomposition of variance|

* Analysis of variance
* Analysis of covariance
* Multivariate AOV

Model exploration|

* Stepwise regression
* Model selection
* Mallows's Cp
* AIC
* BIC
* Model specification
* Regression validation

Background|

* Mean and predicted response
* Gauss–Markov theorem
* Errors and residuals
* Goodness of fit
* Studentized residual
* Minimum mean-square error

Design of experiments|

* Response surface methodology
* Optimal design
* Bayesian design

Numerical approximation|

* Numerical analysis
* Approximation theory
* Numerical integration
* Gaussian quadrature
* Orthogonal polynomials
* Chebyshev polynomials
* Chebyshev nodes

Applications|

* Curve fitting
* Calibration curve
* Numerical smoothing and differentiation
* System identification
* Moving least squares

* Regression analysis category
* Statistics category
*   * Statistics outline
* Statistics topics

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Total least squares. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Total_least_squares&oldid=4348749"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template