Censoring (statistics)

Short description: Condition in which the value of a measurement or observation is only partially known

In statistics, censoring is a condition in which the value of a measurement or observation is only partially known.

For example, suppose a study is conducted to measure the impact of a drug on mortality rate. In such a study, it may be known that an individual's age at death is at least 75 years (but may be more). Such a situation could occur if the individual withdrew from the study at age 75, or if the individual is currently alive at the age of 75.

Censoring also occurs when a value occurs outside the range of a measuring instrument. For example, a bathroom scale might only measure up to 140 kg. If a 160-kg individual is weighed using the scale, the observer would only know that the individual's weight is at least 140 kg.

The problem of censored data, in which the observed value of some variable is partially known, is related to the problem of missing data, where the observed value of some variable is unknown.

Censoring should not be confused with the related idea truncation. With censoring, observations result either in knowing the exact value that applies, or in knowing that the value lies within an interval. With truncation, observations never result in values outside a given range: values in the population outside the range are never seen or never recorded if they are seen. Note that in statistics, truncation is not the same as rounding.

## Contents

* 1 Types
* 2 Analysis
* 2.1 Epidemiology
* 2.2 Operating life testing
* 2.3 Censored regression
* 2.4 Likelihood
* 2.4.1 Example
* 3 See also
* 4 References
* 5 Further reading
* 6 External links

## Types

* Left censoring - a data point is below a certain value but it is unknown by how much.
* Interval censoring - a data point is somewhere on an interval between two values.
* Right censoring - a data point is above a certain value but it is unknown by how much.
* Type I censoring occurs if an experiment has a set number of subjects or items and stops the experiment at a predetermined time, at which point any subjects remaining are right-censored.
* Type II censoring occurs if an experiment has a set number of subjects or items and stops the experiment when a predetermined number are observed to have failed; the remaining subjects are then right-censored.
* Random (or non-informative) censoring is when each subject has a censoring time that is statistically independent of their failure time. The observed value is the minimum of the censoring and failure times; subjects whose failure time is greater than their censoring time are right-censored.

Interval censoring can occur when observing a value requires follow-ups or inspections. Left and right censoring are special cases of interval censoring, with the beginning of the interval at zero or the end at infinity, respectively.

Estimation methods for using left-censored data vary, and not all methods of estimation may be applicable to, or the most reliable, for all data sets.[1]

A common misconception with time interval data is to class as left censored intervals where the start time is unknown. In these cases we have a lower bound on the time interval, thus the data is right censored (despite the fact that the missing start point is to the left of the known interval when viewed as a timeline!).

## Analysis

Special techniques may be used to handle censored data. Tests with specific failure times are coded as actual failures; censored data are coded for the type of censoring and the known interval or limit. Special software programs (often reliability oriented) can conduct a maximum likelihood estimation for summary statistics, confidence intervals, etc.

### Epidemiology

One of the earliest attempts to analyse a statistical problem involving censored data was Daniel Bernoulli's 1766 analysis of smallpox morbidity and mortality data to demonstrate the efficacy of vaccination.[2] An early paper to use the Kaplan–Meier estimator for estimating censored costs was Quesenberry et al. (1989),[3] however this approach was found to be invalid by Lin et al.[4] unless all patients accumulated costs with a common deterministic rate function over time, they proposed an alternative estimation technique known as the Lin estimator.[5]

### Operating life testing

Example of five replicate tests resulting in four failures and one suspended time resulting in censoring.

Reliability testing often consists of conducting a test on an item (under specified conditions) to determine the time it takes for a failure to occur.

* Sometimes a failure is planned and expected but does not occur: operator error, equipment malfunction, test anomaly, etc. The test result was not the desired time-to-failure but can be (and should be) used as a time-to-termination. The use of censored data is unintentional but necessary.
* Sometimes engineers plan a test program so that, after a certain time limit or number of failures, all other tests will be terminated. These suspended times are treated as right-censored data. The use of censored data is intentional.

An analysis of the data from replicate tests includes both the times-to-failure for the items that failed and the time-of-test-termination for those that did not fail.

### Censored regression

An earlier model for censored regression, the tobit model, was proposed by James Tobin in 1958.[6]

### Likelihood

The likelihood is the probability or probability density of what was observed, viewed as a function of parameters in an assumed model. To incorporate censored data points in the likelihood the censored data points are represented by the probability of the censored data points as a function of the model parameters given a model, i.e. a function of CDF(s) instead of the density or probability mass.

The most general censoring case is interval censoring: Pr(a<x⩽b)=F(b)−F(a), where F(x) is the CDF of the probability distribution, and the two special cases are:

* left censoring: Pr(−∞<x⩽b)=F(b)−F(−∞)=F(b)−0=F(b)=Pr(x⩽b)

* right censoring: Pr(a<x⩽∞)=F(∞)−F(a)=1−F(a)=1−Pr(x⩽a)=Pr(x>a)

For continuous probability distributions: Pr(a<x⩽b)=Pr(a<x<b)

#### Example

Suppose we are interested in survival times, T1,T2,...,Tn, but we don't observe Ti for all i. Instead, we observe

(Ui,δi), with Ui=Ti and δi=1 if Ti is actually observed, and
(Ui,δi), with Ui<Ti and δi=0 if all we know is that Ti is longer than Ui.

When Ti>Ui,Ui is called the censoring time.[7]

If the censoring times are all known constants, then the likelihood is

L=∏i,δi=1f(ui)∏i,δi=0S(ui)

where f(ui) = the probability density function evaluated at ui,

and S(ui) = the probability that Ti is greater than ui, called the survival function.

This can be simplified by defining the hazard function, the instantaneous force of mortality, as

λ(u)=f(u)/S(u)

f(u)=λ(u)S(u).

Then

L=∏iλ(ui)δiS(ui).

For the exponential distribution, this becomes even simpler, because the hazard rate, λ, is constant, and S(u)=exp⁡(−λu). Then:

L(λ)=λkexp⁡(−λ∑ui),

where k=∑δi.

From this we easily compute λ^, the maximum likelihood estimate (MLE) of λ, as follows:

l(λ)=log⁡(L(λ))=klog⁡(λ)−λ∑ui.

Then

dl/dλ=k/λ−∑ui.

We set this to 0 and solve for λ to get:

λ^=k/∑ui.

Equivalently, the mean time to failure is:

1/λ^=∑ui/k.

This differs from the standard MLE for the exponential distribution in that the any censored observations are considered only in the numerator.

## See also

* Data analysis
* Detection limit
* Imputation (statistics)
* Inverse probability weighting
* Sampling bias
* Saturation arithmetic
* Survival analysis
* Winsorising

## References

1. ↑ Helsel, D. (2010). "Much Ado About Next to Nothing: Incorporating Nondetects in Science". Annals of Occupational Hygiene 54 (3): 257–262. doi:10.1093/annhyg/mep092. PMID 20032004.
2. ↑ Bernoulli, D. (1766). "Essai d'une nouvelle analyse de la mortalité causée par la petite vérole". Mem. Math. Phy. Acad. Roy. Sci. Paris, reprinted in Bradley (1971) 21 and Blower (2004)
3. ↑ Quesenberry, C. P. Jr. et al. (1989). "A survival analysis of hospitalization among patients with acquired immunodeficiency syndrome". American Journal of Public Health 79 (12): 1643–1647. doi:10.2105/AJPH.79.12.1643. PMID 2817192.
4. ↑ Lin, D. Y. et al. (1997). "Estimating medical costs from incomplete follow-up data". Biometrics 53 (2): 419–434. doi:10.2307/2533947. PMID 9192444.
5. ↑ Wijeysundera, H. C. et al. (2012). "Techniques for estimating health care costs with censored data: an overview for the health services researcher". ClinicoEconomics and Outcomes Research 4: 145–155. doi:10.2147/CEOR.S31552. PMID 22719214.
6. ↑ Tobin, James (1958). "Estimation of relationships for limited dependent variables". Econometrica 26 (1): 24–36. doi:10.2307/1907382. http://cowles.yale.edu/sites/default/files/files/pub/d00/d0003-r.pdf.
7. ↑ , Wikidata Q98961801.

## Further reading

* Blower, S. (2004), D, Bernoulli's ""An attempt at a new analysis of the mortality caused by smallpox and of the advantages of inoculation to prevent it". http://www.semel.ucla.edu/sites/all/files/biomedicalmodeling/pdf/Bernoulli&Blower.pdf. (146 KiB)", Reviews of Medical Virology, 14: 275–288
* Bradley, L. (1971). Smallpox Inoculation: An Eighteenth Century Mathematical Controversy. Nottingham. ISBN 0-902031-23-6.
* Mann, N. R. (1975). Methods for Statistical Analysis of Reliability and Life Data. New York: Wiley. ISBN 047156737X. https://archive.org/details/methodsforstatis00mann.
* Bagdonavicius, V., Kruopis, J., Nikulin, M.S. (2011),"Non-parametric Tests for Censored Data", London, ISTE/WILEY,ISBN 9781848212893.

## External links

* "Engineering Statistics Handbook", NIST/SEMATEK, [1]

* v
* t
* e

Statistics

* Outline
* Index

Descriptive statistics

| Continuous data|

| Center|

* Mean
* arithmetic
* geometric
* harmonic
* Median
* Mode

|

Dispersion|

* Variance
* Standard deviation
* Coefficient of variation
* Percentile
* Range
* Interquartile range

Shape|

* Central limit theorem
* Moments
* Skewness
* Kurtosis
* L-moments

Count data|

* Index of dispersion

Summary tables|

* Grouped data
* Frequency distribution
* Contingency table

Dependence|

* Pearson product-moment correlation
* Rank correlation
* Spearman's rho
* Kendall's tau
* Partial correlation
* Scatter plot

Graphics|

* Bar chart
* Biplot
* Box plot
* Control chart
* Correlogram
* Fan chart
* Forest plot
* Histogram
* Pie chart
* Q–Q plot
* Run chart
* Scatter plot
* Stem-and-leaf display
* Radar chart
* Violin plot

Data collection

| Study design|

* Population
* Statistic
* Effect size
* Statistical power
* Optimal design
* Sample size determination
* Replication
* Missing data

|

Survey methodology|

* Sampling
* stratified
* cluster
* Standard error
* Opinion poll
* Questionnaire

Controlled experiments|

* Scientific control
* Randomized experiment
* Randomized controlled trial
* Random assignment
* Blocking
* Interaction
* Factorial experiment

Adaptive Designs|

* Adaptive clinical trial
* Up-and-Down Designs
* Stochastic approximation

Observational Studies|

* Cross-sectional study
* Cohort study
* Natural experiment
* Quasi-experiment

Statistical inference

| Statistical theory|

* Population
* Statistic
* Probability distribution
* Sampling distribution
* Order statistic
* Empirical distribution
* Density estimation
* Statistical model
* Model specification
* Lp space
* Parameter
* location
* scale
* shape
* Parametric family
* Likelihood (monotone)
* Location–scale family
* Exponential family
* Completeness
* Sufficiency
* Statistical functional
* Bootstrap
* U
* V
* Optimal decision
* loss function
* Efficiency
* Statistical distance
* divergence
* Asymptotics
* Robustness

|

Frequentist inference|

| Point estimation|

* Estimating equations
* Maximum likelihood
* Method of moments
* M-estimator
* Minimum distance
* Unbiased estimators
* Mean-unbiased minimum-variance
* Rao–Blackwellization
* Lehmann–Scheffé theorem
* Median unbiased
* Plug-in

|

Interval estimation|

* Confidence interval
* Pivot
* Likelihood interval
* Prediction interval
* Tolerance interval
* Resampling
* Bootstrap
* Jackknife

Testing hypotheses|

* 1- & 2-tails
* Power
* Uniformly most powerful test
* Permutation test
* Randomization test
* Multiple comparisons

Parametric tests|

* Likelihood-ratio
* Score/Lagrange multiplier
* Wald

Specific tests|

* Z-test (normal)
* Student's t-test
* F-test

Goodness of fit|

* Chi-squared
* G-test
* Kolmogorov–Smirnov
* Anderson–Darling
* Lilliefors
* Jarque–Bera
* Normality (Shapiro–Wilk)
* Likelihood-ratio test
* Model selection
* Cross validation
* AIC
* BIC

Rank statistics|

* Sign
* Sample median
* Signed rank (Wilcoxon)
* Hodges–Lehmann estimator
* Rank sum (Mann–Whitney)
* Nonparametric anova
* 1-way (Kruskal–Wallis)
* 2-way (Friedman)
* Ordered alternative (Jonckheere–Terpstra)

Bayesian inference|

* Bayesian probability
* prior
* posterior
* Credible interval
* Bayes factor
* Bayesian estimator
* Maximum posterior estimator

* Correlation
* Regression analysis

| Correlation|

* Pearson product-moment
* Partial correlation
* Confounding variable
* Coefficient of determination

|

Regression analysis|

* Errors and residuals
* Regression validation
* Mixed effects models
* Simultaneous equations models
* Multivariate adaptive regression splines (MARS)

Linear regression|

* Simple linear regression
* Ordinary least squares
* General linear model
* Bayesian regression

Non-standard predictors|

* Nonlinear regression
* Nonparametric
* Semiparametric
* Isotonic
* Robust
* Heteroscedasticity
* Homoscedasticity

Generalized linear model|

* Exponential families
* Logistic (Bernoulli) / Binomial / Poisson regressions

Partition of variance|

* Analysis of variance (ANOVA, anova)
* Analysis of covariance
* Multivariate ANOVA
* Degrees of freedom

Categorical / Multivariate / Time-series / Survival analysis

| Categorical|

* Cohen's kappa
* Contingency table
* Graphical model
* Log-linear model
* McNemar's test

|

Multivariate|

* Regression
* Manova
* Principal components
* Canonical correlation
* Discriminant analysis
* Cluster analysis
* Classification
* Structural equation model
* Factor analysis
* Multivariate distributions
* Elliptical distributions
* Normal

Time-series|

| General|

* Decomposition
* Trend
* Stationarity
* Seasonal adjustment
* Exponential smoothing
* Cointegration
* Structural break
* Granger causality

|

Specific tests|

* Dickey–Fuller
* Johansen
* Q-statistic (Ljung–Box)
* Durbin–Watson
* Breusch–Godfrey

Time domain|

* Autocorrelation (ACF)
* partial (PACF)
* Cross-correlation (XCF)
* ARMA model
* ARIMA model (Box–Jenkins)
* Autoregressive conditional heteroskedasticity (ARCH)
* Vector autoregression (VAR)

Frequency domain|

* Spectral density estimation
* Fourier analysis
* Wavelet
* Whittle likelihood

Survival|

| Survival function|

* Kaplan–Meier estimator (product limit)
* Proportional hazards models
* Accelerated failure time (AFT) model
* First hitting time

|

Hazard function|

* Nelson–Aalen estimator

Test|

* Log-rank test

Applications

| Biostatistics|

* Bioinformatics
* Clinical trials / studies
* Epidemiology
* Medical statistics

|

Engineering statistics|

* Chemometrics
* Methods engineering
* Probabilistic design
* Process / quality control
* Reliability
* System identification

Social statistics|

* Actuarial science
* Census
* Crime statistics
* Demography
* Econometrics
* Jurimetrics
* National accounts
* Official statistics
* Population statistics
* Psychometrics

Spatial statistics|

* Cartography
* Environmental statistics
* Geographic information system
* Geostatistics
* Kriging

* Category
* '
* Commons
* WikiProject

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Censoring (statistics). Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Censoring_(statistics)&oldid=38992"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template