KACD
Articles
- Page Path
- HOME > Restor Dent Endod > Volume 44(1); 2019 > Article
- Open Lecture on Statistics Statistical notes for clinical researchers: simple linear regression 3 – residual analysis
-
Hae-Young Kim
-
Restor Dent Endod 2019;44(1):e11.
DOI: https://doi.org/10.5395/rde.2019.44.e11
Published online: February 12, 2019
Department of Health Policy and Management, College of Health Science, and Department of Public Health Science, Graduate School, Korea University, Seoul, Korea. JFIF d d Ducky d qhttp://ns.adobe.com/xap/1.0/ |er˭|I|-fGK.rΛp8XV1%K6mvG+tc+qE &ǸC_Nm:l=_/m5^[dߌڇ.c<%:)tQ$Ow~-aY;UJ >=F)2[nk؆?훐 M=l6[4(O.]2#-H^n#->&mp5~Fӛ+|| S,xag% qkEUzUgæBhߕP(7]kFnq?֖Cpru Z6*rEڊtS|*tI*E}7R<,nUU֫^I7Q*mSly%rdȓd8hE<9oHhMfNSRj[i7D[Rj݊+kდq{"$$H?p \̅S?㭻;t~ R߁ )^/>Qj`yt[w ԛ;² ~+ ߔ_ YW~|o] ?x^ᯛ `ʼn;g)T@vWn]>&4l p+$D̢1l|ȨF%-} .9[}w~
ԠLM9hСablfe&QoW !s ?wjLK?s7yO>(=C~_nyǜu? v 3vyo oI@qV-jeES^[9WoSܝh"l2C1a͔CiJ@3:Pճw= /7ovuk+\V;lDgն <[A+r X~d;m!_s8ݖ;;.0llUC+?i#_crʙ1~C.\q u l8Hܶ2m`ܻM3Tov|Bs rɵ"oLS-
DКw=Tv@f '6|YlD͓Y%-#Ѯo%:&!3o%\J<02;K87>^vgƓ# ;ݝmz^Y6=PS39U%~
&f#
}o!muH; ʲŇ˷yvP+&.7e[3'vR4Yj̗IZ`e˽3o[WU{
m[sUbۋZǾۆl6~9'V*.\S2<Sd*zY[aŶ`]C$n.v^Ʌ
dng>ەZ,Mmϑ :n6nϦezW qUJ4! ۇ4R! =>>F n|Q[{pRO 17ƕ~._I ''00k=b՛o}Osðc2'o\3}ݭQ^ 2 . R1yKȣtAݿ-u ܾw!`?1Whn|gzU o[ECWwjUIן)^h#1ɭ!/Z
np;o;ΗŻkXs."6E`Z1
עӐ9Kl8qd q} 2Stt;#j>;խabONŗ=fwP1j)l6J̶|gV 2`y/0E˛6+ԫ1?
6}KW c\KoKͨ2ۅFws*TԞLיuDx .kCzWXhy۶gLu|%TnupǺl-S* P RaLnT+c+*xl.v!.U=|; !_L̎뱚U=4hm:ٯ"y)$:>%(n}X'p[ȴ
^˒4kƓmzDx
\
'Nq amP7nyN݅=
- Correspondence to Hae-Young Kim, DDS, PhD. Professor, Department of Health Policy and Management, Korea University College of Health Science, and Department of Public Health Science, Korea University Graduate School, 145 Anam-ro, Seongbuk-gu, Seoul 02841, Korea. kimhaey@korea.ac.kr
Copyright © 2019. The Korean Academy of Conservative Dentistry
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (https://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
- 3,609 Views
- 75 Download
- 22 Crossref
In the previous sections, simple linear regression (SLR) 1 and 2, we developed a SLR model and evaluated its predictability. To obtain the best fitted line the intercept and slope were calculated by using the least square method. Predictability of the model was assessed by the proportion of the explained variability among the total variation of the response variable. In this session, we will discuss four basic assumptions of regression models for justification of the estimated regression model and residual analysis to check them.
FITTED REGRESSION LINE AND RANDOM ERROR
Let's recall a bivariate relationship between 2 variables, X and Y, and its depiction of a SLR model in Figure 1A and 1B from the previous section [1]. The expression of the regression model is Y=β0+β1X+ε. The regression model is divided into two parts, the fitted regression line, ‘β0+β1X’ and random error, ‘ε.’ The need of error term is justified by the gap between the line and observed dots because regression line does not go through all the observed values as appeared in Figure 1A.
Figure 1
(A) Description of relationship of two variables, X and Y, and error term is expressed as ‘residual’; (B) A single linear regression model; (C) Fitted line and extracted random error part (conceptual).
The first part, fitted regression line, is made by connection of the expected means of Y values corresponding with X values such as χ1 to χ5 in Figure 1B. Please find the conceptual distribution of Y, which corresponds with subgroup of χ1, lying on upper vertical direction. The distribution is displayed as a bell-shaped normal distribution with a mean, μy|χ1, at the center. The symbol, μy|χ1 means expected population mean of Y when X variable has the value χ1. In accordance with the previous sections on regression, the expected mean of Y can be symbolized as Ŷ, which is equal to the fitted line, β0+β1X’.
The expected population mean of Y changes from μy|χ1 to μy|χ5 as X changes from χ1 to χ5. The conceptual model suggests that the expected mean of Y can be depicted as the straight line ‘β0+β1X’ by connecting the expected mean of Y matched with subgroups of X. We call the straight line as ‘mean function’ because the expected mean of Y is expressed as the function of ‘β0+β1X’. Please clearly understand that there are numerous means by numerous subgroups of continuous X, and they are linearly connected to make the linear mean function, ‘β0+β1X’. Therefore, we should be able to reasonably assume that the mean function of Y has the form of fitted regression line when we apply the SLR model.
Now let's discuss the second part, random error ‘ε.’ The conceptual form of the random error is depicted as bell-shaped distributions in Figure 1B. At the center of each distribution, there is the expected mean of Y conditioned on each value of X. As the whole model includes both fitted line and random error part, we can get the random error part by subtracting the fitted line, which is the collection of the expected conditional means, e.g., μy|χ1,..., μy|χ5. What would remain after removing the conditional means? Like Figure 1C, after removing means, the mean of difference in all the subgroups will be changed to zero while the shape of distribution will remain unchanged. There will be an identical distribution of random errors with mean of zeros for all the values of X.
Generally, it is reasonable that we assume the shape of random errors as normal distribution because a small amount of errors can occur frequently, while a large amount of errors may be found rarely. Therefore, traditionally, the error terms are assumed as following distributions: ε~N(0,σ2). The equation means that the distribution of errors follows the normal distribution with mean of zero and variance of a constant, σ2. What does the constant variance tell us? The constant variance shows that all the error distribution for all the subgroups have the same variance. In other words, the shape of the error terms is the same for all the subgroups as shown in Figure 1C.
Wrapping up the discussion above, the assumption of fitted line and random error can be collectively summarized into the distribution of Y. Since Y=β0+β1X+ε, the distribution of Y for each subgroup is Y~N(β0+β1X, σ2) for a given X. According to the definition, the distribution of subgroup for the given value of χ1 is a normal distribution with the mean of β0+β1χ1 and with the constant variance of σ2.
How is the error term expressed in the actual data in Figure 1A? The error term is visualized as ‘observed residuals’, which are the distance between the estimated mean of Y (Ŷi) which locates on the line, and the observed value Yi for the ith observation in Figure 1A. An observed residual, ei, is represented as Yi−Ŷi. If the linear regression model is correctly applied to the observed data, the observed errors from the actual data should be in accordance with the assumption on the distribution of random error. In addition, if the fitted line correctly represents the mean response, the means of residuals for all the subgroups should be near zero and the shapes of residual distributions by subgroups follow the assumed normal distribution, similar to the conceptual error distribution in Figure 1C.
The sum and mean of the observed residuals always equal zero. Suppose the mean of observed residuals is a non-zero value. Then, in calculation procedure for coefficients, the nonzero value should be added into the intercept of the fitted line immediately. Similarly, if the slope of observed residuals is nonzero, the relationship should be added into the slope of the regression line. Eventually, the mean of the observed residuals should be zero and also, the line going through the center of residual distribution should be flat with the slope of zero.
FOUR BASIC ASSUMPTIONS AND RESIDUALS
Four basic assumptions of linear regression are linearity, independence, normality, and equality of variance [2]. Only under the condition that the assumptions are satisfied, the estimated linear line can successfully represent the expected mean value of Y variables corresponding to X values. Also if the assumptions are true, the observed residuals should behave in a similar manner. Let's discuss their meanings in detail.
Linearity means that the means of subpopulation of Y all lie on the same straight line. The scatter plot of X and Y should show a linear tendency. The fitted line of SLR reflects the trend of linearity in the form of linear equation. The error part is expressed as scattered dots around the fitted line as seen in Figure 2A. If the relationship of X and Y is truly a linear one, the scatted dots do not have any trend such as linear, curved, etc., which means that residual and X variable are unrelated. In other words, the residuals appear randomly scattered in relation to X. Also, most observations should lie near the regression line, while observations far away from the line are less frequent, according to the characteristic of assumed normal distribution.
Figure 2
(A) Description of a true linear relationship of two variables, X and Y; (B) Plot of residuals and X from Figure 2A; (C) A mistaken application of linear regression line on squared relationship; (D) Plot of residuals and X from Figure 2C; (E) Plot of residuals and X after adding the true curved relation in the model.
Figure 2A and 2B display a linear relationship of X and Y and a scatterplot of residuals for X values. The residuals locate around zero and are randomly scattered without any patterns. The random feature around zero in residual distribution confirms that the assumption of linearity is correct. In contrast, Figure 2C and 2D come from a simulated curved linear relationship. The original relationship is an equation with linear and squared terms:
Figure 2C shows that by mistake, a linear regression is applied on the data, resulting in the first-order linear equation extracted as Ŷ=−644.7+37.8X. After the linear relation is removed as the fitted line, the distribution of residuals in Figure 2D has a clear pattern of second-order curvature. Therefore, we can presume that there is an omitted term of squared-X and also there is a need to add the squared-X term in the model. After adding the squared term, the residuals finally get to be randomly scattered without any trend (Figure 2E).
Independency assumption means that there is no dependency among observations in the data. In other words, outcome of an arbitrarily selected subject does not affect any other outcomes. In traditional statistics, independence among observations was basically assumed, because the simple random sampling procedure guarantees independence of observations. When we collect data only at one-time point, as a cross-sectional data, we do not worry about independence generally. If sample is chosen by a random sampling method and it is a cross-sectional data, we may simply assume independence and do not actually check this further. However, if the sample was selected using cluster sampling method and there are clusters of subjects in the data, we need to consider the dependence in the analysis procedure.
Violation against this independence assumption frequently occurs in most longitudinal data, which is collected repeatedly from the same subject over time. The repeated data tend to correlate with each other. If we observed subjects in time order, we need to plot those against time order (Figure 3A). In many longitudinal data, each successive error may be associated, not independent. We call the phenomenon ‘autocorrelation’. Sometimes the residuals look like cyclic manners between positive and negative error values, which is called ‘snaking plots’ (Figure 3B).
Figure 3
Linear regression line (A) and error plots (B) in longitudinal data showing ‘autocorrelation’.
The normality assumption is that the distribution of subpopulation is conceptually normal as shown in Figure 1B. How can we confirm this on the observed residuals? If each error distribution of subpopulation is normal, the collective distribution of residuals should be approximately normal.
We generally use either histogram of residuals or normal quantile-quantile plot (Q-Q plot) to check the normality of the distribution. As a kind of Q-Q plots, a normal percentile-percentile (P-P) plot may be used. The histogram of residuals should appear similar to normal. In Q-Q plots, the observed points (dots) should be around the diagonal line (Figure 4). Plotting standardized residuals can give us more information to check the size of residuals.
Figure 4
Residual plots to check the normality assumption. (A) Histogram of standardized residuals. (B) Normal percentile-percentile (P-P) plot.
Cum Prob, cumulative probability.
Conceptually, the error terms of all the subgroups are assumed to have the same shape of distribution. That is the normal distribution with mean of zero and variance of a constant, σ2, as shown in Figure 1C. In SLR, we can plot observed residuals against X, because the fitted value and X are linearly related, as shown in Figure 1C. However, in multiple linear regression with more than one X, we need to consider a scatter plot of residuals and fitted values, Ŷ, which is a linear combination of Xs.
The equal variance assumption means that the degree of spreading or variability of residuals is equal across the subpopulations with different fitted values. We call this feature ‘homoscedasticity,’ which means having the same variance. Figure 5A and 5B do not show any definite violence of the equal variance, because the length of orange dotted lines in Figure 5B are not considerably different. The scattered residual plot against Ŷ displays an approximate squared form around zero.
Figure 5
Examples of homoscedasticity and heteroscedasticity. (A, B) Residual plot against fitted value of Y (or X) showing homoscedasticity; (C) Relationship of X and Y; (D) Residual plot against fitted value of Y (or X) showing heteroscedasticity.
In contrast, Figure 5C and 5D show that the variability is increasing as the value of X or Ŷ increases. We call the phenomenon of unequal variances ‘heteroscedasticity’. As shown in Figure 5C, increasing variability by increase of mean is not a rare phenomenon. For example, let's consider the relationship between income and expenditure. We can easily imagine that the variability of expenditure will increase among people with high income, because they can choose high amount or low amount of expenditure according to their propensity to consume, while poor people can choose only among a limited amount of expenditure because of lack of income. The relationship violates the assumption of homoscedasticity.
As summary, we can use residual plots to check three basic assumptions of linear regression as following:
- Linearity: (standardized) Residuals against X variable.
- Normality: Histogram of (standardized) residuals, Normal Q-Q (or P-P) plot of (standardized) residuals.
- Equal variance: (standardized) Residuals against Ŷ (or X only in SLR) variable.
REFERENCES
- Appendix 1
- D-1: During regression procedure in SLR [1], residual plots against Ŷ (ZRESID*ZPRED) to check equal variance assumption and standardized residual plots (histogram and normal probability plot) to check normality assumption are requested.
- D-2: To save the predicted values and residual values (original & standardized). Standardized residual values larger than 2 in absolute scale can be used to identify outliers.
- H-1 and H-2: To request a scatter plot of (standardized) residual against X to check the linearity assumption.
Appendix
Procedure of residual analysis of simple linear regression (SLR) model using IBM SPSS
The residual analysis of SLR using IBM SPSS Statistics for Windows Version 23.0 (IBM Corp., Armonk, NY, USA) is performed during or after regression procedure appeared in the previous session [1].
aTo request scatter plot (F), and the histogram and the normal p-p plot in Figure 4, bScatter plot of standardized residuals and predicted value to check equal variance assumption, cTo save predicted values (PRE_1), residual (RES_1), and standardized residuals (ZRE_1), and dScatter plot to check linearity assumption.
Tables & Figures
REFERENCES
Citations
Citations to this article as recorded by 
- Anomaly detection using different types of machine learning models in the context of “smart maintenance technologies in the manufacturing industry”
San Giliyana
Procedia Computer Science.2025; 253: 942. CrossRef - Investigation of the effect of flocculation and pulp parameters on the performance of vacuum filtration of Moroccan phosphate slurry using novel definitive screening design
Ahmed Alalou, Zouhair Hafid, Bouali Ali, El Yazghi Ezzaher Amine, Mohammed El Asri, Ahmed Boulahna
Chemical Papers.2025; 79(8): 5139. CrossRef - A Scale Development Study on the Expectation of a Disabled Friendly Hotel in the Context of Accessible Tourism
Ayşe Himmetoğlu, Beril Erkaya, Zeki Akıncı, Gülseren Yurcu, Edina Ajanovic, Gürkan Aybek, Didem Kutlu
Advances in Hospitality and Tourism Research (AHTR).2025;[Epub] CrossRef - Clinically significant bradycardia in children with respiratory syncytial virus bronchiolitis receiving dexmedetomidine: effect modification by mechanical ventilation
Rasha S Farag, Aditya S Kalluri, Geetha Iyer, Jennifer P Stevens, Carly E Milliren, James Brian McAlvin
BMJ Paediatrics Open.2025; 9(1): e003625. CrossRef - Evaluation of the influence of mixtures of nonionic surfactants on the rheology of bituminous coal-water slurry using simplex centroid mixture design
Ahmed Alalou, Abiodun A. Saka, Mohammed El Asri, Ahmed Boulahna, Monday U. Okoronkwo, Muthanna H. Al-Dahhan
Journal of Dispersion Science and Technology.2025; : 1. CrossRef - Investigating the effect of mixtures of anionic surfactants on the potential stability and rheology of coal-water slurry using the mixture design methodology
Ahmed Alalou, Abiodun A. Saka, Mohammed El Asri, Ahmed Boulahna, Monday U. Okoronkwo, Muthanna H Al-Dahhan
International Journal of Coal Preparation and Utilization.2025; : 1. CrossRef - A usability study on mobile EMG-guided wrist extension training in subacute stroke patients-MyoGuide
Hao-Ping Lin, Yang Xu, Xue Zhang, Daniel Woolley, Lina Zhao, Weidi Liang, Mengdi Huang, Hsiao-ju Cheng, Lixin Zhang, Nicole Wenderoth
Journal of NeuroEngineering and Rehabilitation.2024;[Epub] CrossRef - Prediction of high-performance concrete strength using machine learning with hierarchical regression
Iman Kattoof Harith, Wissam Nadir, Mustafa S. Salah, Mohammed L. Hussien
Multiscale and Multidisciplinary Modeling, Experiments and Design.2024; 7(5): 4911. CrossRef - Influence of the radius of Monson’s sphere and excursive occlusal contacts on masticatory function of dentate subjects
Dominique Ellen Carneiro, Luiz Ricardo Marafigo Zander, Carolina Ruppel, Giancarlo De La Torre Canales, Rubén Auccaise-Estrada, Alfonso Sánchez-Ayala
Archives of Oral Biology.2024; 159: 105879. CrossRef - Fish oil supplementation modifies the associations between genetically predicted and observed concentrations of blood lipids: a cross-sectional gene-diet interaction study in UK Biobank
Yitang Sun, Tryggvi McDonald, Abigail Baur, Huifang Xu, Naveen Brahman Bateman, Ye Shen, Changwei Li, Kaixiong Ye
The American Journal of Clinical Nutrition.2024; 120(3): 540. CrossRef - Associations between Dietary Sugar and Fiber with Infant Gut Microbiome Colonization at 6 Mo of Age
Pari Mokhtari, Elizabeth A. Holzhausen, Bridget N. Chalifour, Kelsey A. Schmidt, Mahsa Babaei, Christopher J. Machle, Shana Adise, Tanya L. Alderete, Michael I. Goran
The Journal of Nutrition.2024; 154(1): 152. CrossRef - A machine-learned model for predicting weight loss success using weight change features early in treatment
Farzad Shahabi, Samuel L. Battalio, Angela Fidler Pfammatter, Donald Hedeker, Bonnie Spring, Nabil Alshurafa
npj Digital Medicine.2024;[Epub] CrossRef - Response surface methodology optimization of the effect of pH, contact time, and microbial concentration on chemical oxygen removal potential of vegetable oil industrial effluents
Abiodun Onadeji, Badruddeen Saulawa Sani, Umar Alfa Abubakar
Water Environment Research.2024;[Epub] CrossRef - Selecting, optimizing and externally validating a preexisting machine-learning regression algorithm for estimating waist circumference
Bryan V. Phillips-Farfán
Computers in Biology and Medicine.2024; 169: 107909. CrossRef - QUALITY CONTROL ASSESSMENT OF DUTASTERIDE AND SILODOSIN IN CAPSULES AND TABLETS EMPLOYING A NOVEL DEVELOPED HPLC TECHNIQUE; EVALUATION OF STABILITIES OF DUTASTERIDE AND SILODOSIN IN ACCELERATED DEGRADATION
KADALI JAGADEESH, K. GANESH KADIYALA, B. N. SURESH VARMA DENDUKURI, RAMA SWAMY GUTTULA, V. L. N. BALAJI GUPTA TIRUVEEDHI, PEDDINTI VAMSI, RAJYA LAKSHMI CHAVAKULA
International Journal of Applied Pharmaceutics.2023; : 98. CrossRef - Statistical Estimation of Soil Carbon Stocks in Chungcheong Province through Digital Soil Mapping and Multiple Linear Regression
Yun-Gu Kang, Jae-Han Lee, Jun-Yeong Lee, Taek-Keun Oh
Korean Journal of Soil Science and Fertilizer.2023; 56(3): 209. CrossRef - Modelling of Low-Voltage Varistors’ Responses under Slow-Front Overvoltages
Lutendo Muremi, Pitshou N. Bokoro, Wesley Doorsamy
Electronic Materials.2023; 4(2): 62. CrossRef - Evaluating Changes in Trauma Epidemiology during the COVID-19 Lockdown: Insights and Implications for Public Health and Disaster Preparedness
Mariusz Jojczuk, Jakub Pawlikowski, Piotr Kamiński, Dariusz Głuchowski, Katarzyna Naylor, Jakub Gajewski, Robert Karpiński, Przemysław Krakowski, Józef Jonak, Adam Nogalski, Dariusz Czerwiński
Healthcare.2023; 11(17): 2436. CrossRef - Impact of Regional Development Strategy on the Productivity of Polluting Firms:Evidence From China
Jianmin Dou, Zhipeng Tao, Yongbao Ji
Frontiers in Environmental Science.2022;[Epub] CrossRef - Factors Associated with Japanese Nurses’ Support on End-of-life Decision-making: A Cross-sectional Study
Asahiko Higashitsuji, Tomoko Majima
Journal of Community Health Nursing.2022; 39(3): 139. CrossRef - Postnatal exposure to ambient air pollutants is associated with the composition of the infant gut microbiota at 6-months of age
Maximilian J. Bailey, Elizabeth A. Holzhausen, Zachariah E. M. Morgan, Noopur Naik, Justin P. Shaffer, Donghai Liang, Howard H. Chang, Jeremy Sarnat, Shan Sun, Paige K. Berger, Kelsey A. Schmidt, Frederick Lurmann, Michael I. Goran, Tanya L. Alderete
Gut Microbes.2022;[Epub] CrossRef - Forecasting surgical costs: Towards informed financial consent and financial risk reduction
Savio George Barreto, Norma Bulamu, Adarsh Chaudhary, Gang Chen, Kazuki Kawakami, Laura Maggino, Giuseppe Malleo, Sayali Pendharkar, Maxwell T. Trudeau, Roberto Salvia, Charles M. Vollmer, John A. Windsor
Pancreatology.2021; 21(1): 253. CrossRef
ePub Link
Cite
XML DownloadStatistical notes for clinical researchers: simple linear regression 3 – residual analysis
Figure 1 (A) Description of relationship of two variables, X and Y, and error term is expressed as ‘residual’; (B) A single linear regression model; (C) Fitted line and extracted random error part (conceptual).
Figure 2 (A) Description of a true linear relationship of two variables, X and Y; (B) Plot of residuals and X from Figure 2A; (C) A mistaken application of linear regression line on squared relationship; (D) Plot of residuals and X from Figure 2C; (E) Plot of residuals and X after adding the true curved relation in the model.
Figure 3 Linear regression line (A) and error plots (B) in longitudinal data showing ‘autocorrelation’.
Figure 4 Residual plots to check the normality assumption. (A) Histogram of standardized residuals. (B) Normal percentile-percentile (P-P) plot.Cum Prob, cumulative probability.
Figure 5 Examples of homoscedasticity and heteroscedasticity. (A, B) Residual plot against fitted value of Y (or X) showing homoscedasticity; (C) Relationship of X and Y; (D) Residual plot against fitted value of Y (or X) showing heteroscedasticity.
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Statistical notes for clinical researchers: simple linear regression 3 – residual analysis
