It is really not the size of the intraclass correlation that is the issue. It is the size of the design effect, which is a function of the intraclass correlation and the average cluster size. A design effect greater than 2 indicates that the clustering in the data needs to be taken into account during estimation. The design effect is approximately equal to
My source is personal communication, Bengt Muthen. Seriously, however, Bengt has no reference for this rule of thumb besides his paper with Albert Satorra in Sociological Methodology in 1995. In Table 2, results are reported for a Monte Carlo study. Although design effects are not presented, the approximate design effects can be computed. It seems approximate design effects of less than two do not result in overly exaggerated rejection proportions at the 5 percent level using Method 1. For example, with an intraclass correlation of .05, the approximate design effects for cluster sizes of 7, 15, and 30 are 1.3, 1.7, and 2.45, respectively. As shown in the table the 5 percent reject proportions are satisfactory for 7 and 15 but not for 30. Table 3 gives corresponding results for standard errors.
It would be interesting to hear other people's experiences about which size of design effects are considered important.
I have an inquiry regarding the use of a longitudinal study (NLSY79) that employs a complex sample design (DEFF estimated at 1.5 to 1.3 at various time point). I am testing a model that has a continuous outcome variable, but categorical exogenous (indirect and direct effects) and "mediating" endogeneous variables that are categorical. The exogenous variables are time-bound, i.e., they occur prior to my outcome (net worth in 1996). Does Mplus handle data from complex sample designs and is SEM the best way to proceed?
Mplus does handle data from some complex sample designs for continuous outcome variables. So if you have categorical mediating variables, this would be a problem. However, with your small DEFF's, perhaps you could estimate the model as if it is a random sample. Then Mplus could handle the categorical mediating variables. If needed, you could still use weights in the analysis.
Regarding the appropriateness of SEM for your problem, a growth model of the type you describe can be estimated using SEM. The latent variables are used to represent the growth factors. See the examples section of the website for some examples of growth models.
SUNGWORN posted on Friday, April 07, 2000 - 9:33 pm
I try to study multilevel topic that was proposed by Muthen, but I do not understand the exact meaning of this equation...Sigma T = SigmaW + C*SigmaB. 1. why does he scale SigmaB with C and what is C*SigmaB represent? 2. How does this equation differ from .....Sigma T = SigmaW + SigmaB... in ANOVA ? 3. How does intraclass correlation from ANOVA (ICC = SSB/SST) differ form HLM program.?
These quantities are discussed on page 299 of the Mplus User's Guide. In equation (156) it says that SigmaW + s*SigmaB is the expected value of SB, that is the expected value of the between covariance matrix in the sample. In this equation SigmaB needs to be scaled with the average cluster size s (for further details, see, e.g. Muthen, 1990). SigmaT = SigmaW + SigmaB as usual. The intraclass correlation is defined in equation (160) as the ratio of between and total variances; I would expect this is the same as in HLM.
Anonymous posted on Tuesday, May 15, 2001 - 11:57 am
I am interested in looking at correlations between variables for data that have a multi-level structure (individuals within communities). From my rudimentary understanding of Mplus, I should be able to use Mplus to obtain correlations within communities and estimates of correlations between communities. If this assumption is correct, I have two questions:
1) How does Mplus handle dichotomous variables at the aggregate level? If these variables were coded 1/0, would Mplus sum these values at the aggregate (community) level or average these values?
2) How would I compute the degrees of freedom and test the significance of the correlations at each level (within and between)?
bmuthen posted on Wednesday, May 16, 2001 - 8:30 am
Mplus multilevel modeling currently only handles continuous dependent variables. Dichotomous variables are, however, allowed if they are not dependent variables. Individual-level variables are essentially community-level means at the community level. Significance of correlations on either level can be computed using t ratios of parameter estimates to standard errors.
Anonymous posted on Tuesday, August 20, 2002 - 2:23 pm
I'm confused about the relationship between the Mplus hierarchical SEM and the Mplus options for the analysis of complex sample survey data.
Doesn't hierarchical modeling already take into account design effects, or does it only adjust the parameter estimates but not the SEs of the estimates for clustered data ? I thought conventional HLM analyses inherently take clustering into account.
TYPE=COMPLEX adjusts the standard errors and chi-square fit statistic for clustering but not the parameter estimates. Yes, hierarchical modeling, ie, TYPE=TWOLEVEL, takes clustering into account. TYPE=TWOLEVEL models the clustering and therefore adjusts the parameter estimates, standard errors, and chi-square test statistic.
Anonymous posted on Tuesday, August 20, 2002 - 4:26 pm
To follow up on your reply:
does this mean that the CLUSTER option is redundant in Mplus and does not need to specified when doing a hierarchical SEM analysis ?
No, this is how Mplus knows which clusters individuals belong to so that multilevel modeling can be done.
Anonymous posted on Thursday, September 30, 2004 - 6:44 am
I am doing a 2-level SEM, the latent factor INFOR is measured by five observed indicators (X1-X5). I find that once I change the the fixed value of observed indicators, (simply by switching the order position of measurement indicaotrs, ie. INFOR BY X1-X5, change to INFOR X2 X1 X3 X4 X5)
all model estimates seems indentical except for the variance of latent factor at both levels. As a result, the intraclass correlation of latent factor moves from 27% to 91%. So my quesiton is:
how to solve the problem? which indicator estimated should be fixed to one? is the difference in ICC values of latent factor related to the scale of observed indicators?
I can understand that the variance of the factor will change depending on the item selected for setting the factor's metric but I don't understand why the intraclass correlation would change to dramatically. Can you please send the two outputs and the data to email@example.com?
This invariance of the latent ICC can only be expected when the within loadings are held equal to the between loadings. Otherwise, different factors are considered. With loading invariance, you will find latent ICC invariance.
Jim McMahon posted on Monday, October 11, 2004 - 11:42 am
The rule of thumb value of 2 or greater for the design effect to justify a multilevel model [Oct 29, 1999] is very interesting to me. I am working on multilevel models with dyadic data (2 subjects per group). The ICC would have to be 1.0 in order for the design effect to be 2, according to
The rule of thumb is intended to give a flavor of when one would go wrong in terms of standard errors and chi-square when ignoring the multilevel structure of the data. Multilevel modeling can be done even when there are smaller design effects when the multilevel structure itself is of interest, as it clearly is with dyadic data.
Tom Munk posted on Thursday, November 17, 2005 - 11:48 am
I have been trained to compute ICC as: between variance/(between variance + within variance).
For example, I have a model with: between variance = 140 within variance = 900 my icc calculation = .135
But MPLUS provides an ICC of .300.
How does MPLUS calculate ICC? How should it be interpreted?
Yes, this is how we compute an intraclass correlation. See formula 203 Technical Appendix 10. I would have to see where you get your information to comment further. Please send your input, data, output, and license number to firstname.lastname@example.org.
Marco posted on Wednesday, December 14, 2005 - 8:18 am
Hello Linda, hello Bengt,
in the 1994-paper about MCA, it says that the ICC is a rough indication of between-variation (step 2 of the recommended 5 steps) and that the need of an multilevel analysis could be tested be restricting Sig-B = 0 in an MCA. How would I do the latter and is it worth the effort (compared to simply look at the ICC´s to determine the need of an multilevel strategy)? Many thanks!
bmuthen posted on Wednesday, December 14, 2005 - 8:25 am
You set Sig_B = 0 simply by saying (assuming 2 variables):
I obtain invariant ICCs as well. When fitting an empty 2-group model on a data file I obtain two ICCs (one for each group), but when I specify the regression models on both levels, the dependent variable's ICCs change. Why is that? The regression model should refer to the level specific variances of the dependent variable without changing them. I thought the ICC of a dependent variable to be independent from explanatory variables.
When doing a kind of preface analysis checking the ICCs, I obtain higher values compared to those ICCs from the regression model. Which of those should I report?
Noona Kiuru posted on Thursday, June 15, 2006 - 2:27 am
I have a question concerning relations between intraclass correlations and statistical significance of between-level and within-level variance estimates in the multilevel context. I would like to test whether clusters become more homogeneous across time (two time points) or, alternatively, more differentiated from other clusters. I thought that this could be done by comparing estimates of between-variance in outcome variable between Time 1 and Time 2, on one hand, and estimates of within-variance between Time 1 and 2 on the other. This comparison could be made by constructing a model for the observed variables at Time 1 and Time 2 in which the between- and within level variances of the observed outcome variables are set equal across time. However, one referee suggested that this is not a proper way to study whether the clusters become more homogeneous (or similar). He/she suggested that I should test statistical significance of the difference between the intraclass correlations of two time points (these are .18 and .34, respectively) instead. Do you have any ideas what would be the best way to study this issue (i.e., whether clusters become more homogeneous across time)? And if it should be done by comparing intracluster correlations how should I do this? Thanks
"diff" gives you the estimate and SE for the icc difference. For the model where you set the icc's equal, you simply add diif=0. You can also compare the loglikelihoods for these 2 models in a standard 1 df chi-square test.
David Bard posted on Friday, August 11, 2006 - 12:04 pm
1) I'm aware of kappa coefficients for this, but I'm wondering if it's also appropriate to calculate an ICC for two-level regression models where the dependent variable is categorical. Can you simply divide the sum of the between level residual variances and twice their covariances (i.e., between variability) by the standardized variance of the probit (=1) or logit (=pi/sqt(3), right?) link functions, i.e., the total variance? I suspect I'm inaccurate, b/c the ICCs I've calculated in this fashion change substantially depending on the link function I use. 2) Again for a categorical DV two-level regression model, I've tried modeling the covariance between a random threshold and a random level-1 slope. I get an estimate, but the total between variance changes quite a bit when I include this term (variance doubles without the covariance modeled: .052 to .122). Does it make sense to have this term in the model in the categorical DV case, and should this change in the between-level variability be of concern?
David Bard posted on Friday, August 11, 2006 - 2:16 pm
I've just realized I was using the standard deviation of the standardized logistic distribution when calculating the ICC described above. When I change it to a variance, pi^2/3, the "alleged" ICCs are much more similar across link functions. So I guess my question 1) now is simply whether this calculation is appropriate? Question 2) still stands as is.
1) this is a somewhat reasonable approach I think, much like one can say it is somewhat reasonable to work with an r-square for a continuous latent response variable underlying a categorical dv.
2) yes, it makes sense to have this term included. If the variance of the slope is significant, I would include it. If that makes the results change a lot, perhaps that is because the likelihood is a lot better.
I have data on adolescents' and their best friends' delinquency on five waves and want to look at associations between the growth curve of the target adolescents and growth curve their best friends. So I want to look whether the slope of delinquency of the target adolescent is predicted by the intercept of deliqnuency of the best friend, and vice versa (and the intercept-intercept correlation and slope-slope correlation). What would be the best way to model such associations? I am aware that I am dealing with dyadic data which should be treated with a multilevel method, and therefore should look at intraclass correlations, but at the same time my aim is to look at the strength of the association between the intercepts, the slopes, and the intercept-slope correlations.
What comes to mind when I read your message is the type of analysis suggested in the following paper:
Khoo, S.T. & Muthén, B. (2000). Longitudinal data on families: Growth modeling alternatives. Multivariate Applications in Substance use Research, J. Rose, L. Chassin, C. Presson & J. Sherman (eds.), Hillsdale, N.J.: Erlbaum, pp. 43-78. (#79)
Anup posted on Saturday, February 24, 2007 - 2:28 pm
What is the appropriate formula to calculate design effect for 3-level design? Are the formula presented above still applicable? Since there are 2 ICCs: level 2 and level 3 in such design, is there any adjustment needed in design effect formula 1 + (average cluster size - 1)*intraclass correlation ?
I don't know off hand what the formula is. It is probably covered in the classic Cochran book Sampling Techniques. Note that the formula you cite is for the special case of estimating means and with equal cluster sizes
hello, I'm carry on a multivel analysis, just an empty model for categorical variables. How is possible to obtain the ICC with it's p value and the p value of the estimates (thresholds and variances)? Thanks a lot Regards, Elisa
Boliang Guo posted on Tuesday, March 13, 2007 - 3:56 am
ICC(logistic reg)=(Level2 Var)/(Level2 var+ (phi2/3)) look Snijders's book
elisa posted on Wednesday, March 14, 2007 - 2:35 am
Thanks a lot, but I would like to know if Mplus provides the ICC estimate "automatically", and if Mplus provides also the ICC p-value in order to have a direct test. (I saw in another message posted here that Mplus provides an ICC, but I'm not able to find the command and I calculated it)
If Mplus does not provide the ICC p-value, how is possible to test its significance?
CATEGORICAL ARE responce ; CLUSTER = group; TYPE = twolevel random ; ESTIMATOR = ml; MODEL: %WITHIN%; %BETWEEN% responce;
And, how is possible (if possible) to obtain automatically the p-value of the other estimates (thresholds and variances)?
Boliang Guo posted on Wednesday, March 14, 2007 - 2:50 am
as I know, statistically significant level 2 variance is necessary for Multilevel analysis, ICC provide information of the degree of level 2 variance over total variance. I am using MLwiN, HLM, Mplus and WinBugs, only WinBugs can give uncertainty of ICC. well, I think you get no nore information from ICC than the significance of level 2 variance. I do not know the command for ICC, but I think you can use Model constraint to get ICC in Mplus. maybe Prof Muthen can give more information on this topic.
Yes, you would use MODEL CONSTRAINT to create the ICC by dividing the estimated between variance by the sum of the estimated between and within variances. The program then gives a standard error for this parameter.
I have a multilevel path model, that includes variables X1b (measured at the group level), Y1 and Y2 (measured at the individual level). The between group model suggest that X1b effects Y1b (average Y1 in groups) (X1b -- > Y1b).
The within group portion of the model suggests that Y1 effects Y2 (both at the individual level) (Y1 --> Y2).
I also want to test if Y1 mediate the relationship between X1b (group level) and Y2 (individual level) (X1b --> Y1 --> Y2).
Is this possible in MPlus? If so, how can I develop a syntax, or is there any example that I can use?
Mike Tobak posted on Thursday, June 28, 2007 - 1:10 pm
I wonder if Mplus can automatically produce intraclass correlations as well as between and within group covariance matrix without estimating any specific models.
I just want to check if my data is appropriate for multilevel analysis (I am afraid that the variance of certain variable is close to zero at the second level), and I don't need to test any model at this time.
I just tried a simple model and Mplus couldn't give the estimations due to nonpositive definite matrix. And I want to know if there is any option in Mplus to produce intracorrelations and between-/within-group covariance matrix alone.
Yes, you can use TYPE=BASIC TWOLEVEL; to do this. You can use the SAMPLE and SIGB options of the SAVEDATA command to save the matrices. See the Mplus User's Guide for more information.
Mike Tobak posted on Thursday, July 12, 2007 - 9:26 pm
I have a question about the between- and within- groups matrices. I think the scaled between-group matrix should be larger than pooled within group matrix, since S(between)=Cov(within)+c*Cov(between). However, the scaled between-group matrix calculated from my data (I programmed in Matlab using the given formula) is less than pooled within group matrix, producing negative (but each covariance/variance component is very small, close to zero) between groups matrix. Is there any possibility to have negative (but close to zero) estimated variance for between groups? It sounds weird, but I checked my code, and it works well for other data. Can I say that my data has no multilevel structure? Thanks.
The formula you give is equation (199) in Appendix 10 on the Mplus web site. Note that this formula is for the expected value of SB, not for the sample SB, where the sample SB is defined in (197). Using this expected formula, the estimated SigmaB is given in (202) and yes it can obtain negative values indicating no between variation.
Mike Tobak posted on Sunday, July 15, 2007 - 8:21 pm
Prof. Muthen, Thank you!
I used the correct formula as you said. I noticed that you mentioned (Muthen 1994, multilevel covariance analysis, sociological methods & research, vol.22, No.3, 376-398) that "the ML estimator of SigmaB is frequently not positive definite and might not even have positive variance estimates. This means that, in practice, we might have to resort to analyzing SB to get a notion of the SigmaB structure. Fortunately, experience shows that when it is possible to analyze both matrices, similar results are obtained." (on page 389). Does that mean that even if the ML estimator (given in (202)) has negative variance estimates, there could still be between variations and we can get similar results by analyzing SB?
I am a little confused here. According to your answers to my last question, there should be no between variations if the estimated SigmaB from (202) has negative values.
We know too little of these situations to say anything with certainty. If the SigmaB estimate from (202) has negative values it is likely that there is no between variation in the population, but on the other hand there may be some. The quote from my paper indicates similar results when analyzing Sb and estimated SigmaB - "when it is possible to analyze both matrices". I would say that precludes the case of negative variances for estimated SigmaB. The best way to find out if there is between variation is to fit the full two-level model using FIML.
Mike Tobak posted on Saturday, July 28, 2007 - 1:17 pm
Prof. Muthen, THANK YOU!
I wonder if we can test it by multilevel regression (with random slopes).
For instance, we fit a level-one model with variable y on two other level-one variables x1 and x2, and a level-two model of the random intercepts/slopes on a level-two variable W. We found that the intercepts/slopes from level-one model are indeed random effects (variance is significant). By doing this, can we say that the level-one units are not independent since intercepts/slopes are random effects, even thought we got negative (but close to 0) estimated SigmaB?
In fact, that is what happened to my data. The calculated intraclass correlations are close to 0, but the estimated SigmaB is negative (close to 0 though). Then I use multilevel regression (in HLM or SAS). if I specify an ANOVA model (without any exploratory variables at level-one and level-two), the intercept is not a random effect (hence agree with small intraclass correlations); but if I include some level-one exploratory variables, the random slopes of such exploratory variables are random effects (having significant variance tested at level-two model). Does this make sense to you? Small intraclass correlations but the slopes of exploratory variables are random across different groups?
We compute the intraclass correlations using the unbiased estimates obtained via SAMPSTAT or TYPE=BASIC not the consistent ML estimates from the model.
Mike Todd posted on Tuesday, September 25, 2007 - 5:04 pm
Thanks for getting back to me so quickly, Linda.
It looks like the discrepancy was due to a misinterpretation of the output on my part.
The values I presented above were from a Monte Carlo run, where the sample stats from the first data replication were reported in the output. I (incorrectly) assumed that the accompanying ICC was based on these statistics.
When I took the raw data from the first replication and used them in a "non-Monte Carlo" two-level run, I obtained an ICC that is consistent with the sample statistics (i.e., ICC = 0.292).
Thanks again for your help.
Mike Todd posted on Wednesday, September 26, 2007 - 9:50 am
After sleeping on this, I realized I should have asked what exactly the ICC reported in the Monte Carlo output represents.
Is this described in any of the documentation or online?
They are the intraclass correlations for the first replication in Version 4.21. In earlier versions, they were for the last replication. Is this what you are asking?
Mike Todd posted on Wednesday, September 26, 2007 - 11:56 am
Thanks, Linda. That's exactly what I was asking. The output I found confusing was generated under version 4.1. I just re-ran the Monte Carlo analysis under 4.21 and got the expected result. Thanks again!
I just start using Mplus to see if some models require multilevel CFA, and am new to multilevel analysis. As I put in different numbers of dependent variables into a 2-level model, I find that the ICC of the variables are different. For instance, when I put in 5 DVs, the ICCs for these 5 DV are different when I put in one more DV. All my data do not contain missing value. May I ask the mechanism for this difference? Using other software does not seem to result in such difference.
Sharon posted on Saturday, March 29, 2008 - 11:02 am
I have a question relevant to a posting on May 15, 2001. Like that individual, I am interested in looking at correlations in a multilevel context. To do this, I have been examining the between and within correlation matrices using twolevel analyses. (a) Is it appropriate to interpret correlations involving two continuous variables in the within correlation matrix as the equivalent of standard rs, but corrected for the multi-level nature of the data (e.g., can I square them to get estimates of effect size)? (b) are there any special considerations in interpreting these correlations? (c) I'd like to report these correlations in articles I'm submitting; do you know of any examples where this has been done in the literature (or any problems people have had)?
The within covariance is the covariance between the variables excluding the covariance due to clustering. In terms of correlations, you should decide if you want to calculate the correlations using total variances or within and between variances. Mplus uses within and between variances. I know of no special considerations in the interpretation of the correlations. I am not familiar with squaring a correlation to get effect size.
I have a simple question which I'm having trouble getting an answer to concerning the interpretation of ICCs. Take the following hypothetical example:
100 high schools are randomly sampled, from each of which 100 seniors are randomly sampled. SAT scores of the seniors are recorded. The ICC is found to be a statistically significant .01. Only 1 percent of the total variance in SAT scores is between schools.
Based on this small ICC, would it be wrong to interpret this as follows: "The school a student attends is not particularly relevant to the student's performance on the SAT."
Hi, my understanding of the intraclass correlation for multilevel analysis is that it is like the R-square in the linear regression (i.e., how much variance is explained by the between-cluster variance). However, if the dependent variable is count data, does intraclass have substantive meaning? And, can I obtain it from MPLUS analysis? Thanks!
You can use TYPE=TWOLEVEL BASIC to obtain the information to compute the ICC's for categorical outcomes. We do not do this in Mplus.
Boliang Guo posted on Wednesday, August 27, 2008 - 7:18 am
I remember there is a formula in Prof Muthen's Utrchet lecture. for random intercept only model, the ICC for 2-level logistic regression, ICC(logistic reg)=(Level2 Var)/(Level2 var+ (phi2/3)) look Snijders's book, you pls check Mplus discussion for some answers
Hi, I'm running a simple path analysis in which adolescent and parental negative emotional expression predict adolescent self-regulation. To account for non-independence of my data, I used TYPE=COMPLEX and clustered it by family (805 families). In my model, there is a significant path between my exogenous variables of adolescent negative emotional expression and parental negative emotional expression. Since I've clustered my data by family, is this correlation the same as an interclass correlation? This is my first time using the TYPE=COMPLEX command, so I'm a little hesitant in interpreting my output. thank you for your help!
My first impression is that you account for non-independence twice - once is enough.
Type=Complex would treat family members as members of the cluster=family. If a family member is observed on p variables, there are p observed variables in the model.
If on the other hand you have a parent and an adolescent in each family and each is observed on p variables, there are 2*p variables in the model. When you say that parental expression is regressed on adolescent expression, this is the situation that comes to my mind. In this case, Type = Complex is superflous because the intra-family correlation between parent and adolescent is modeled as you say.
For related work in a growth model framework, see
Khoo, S.T. & Muthén, B. (2000). Longitudinal data on families: Growth modeling alternatives. Multivariate Applications in Substance use Research, J. Rose, L. Chassin, C. Presson & J. Sherman (eds.), Hillsdale, N.J.: Erlbaum, pp. 43-78.
Hi Bengt, Thank you for the quick reply and reference suggestion! May I double-check that I understand your response? My model looked like this: selfreg on mom adol; mom with adol; Since I already clustered my "mom" and "adol" predictors by family using the TYPE=COMPLEX, are you saying that allowing "mom" and "adol" to be correlated in my model is redundant? If that is true, should I assign an meaning to that correlation in my model, given that it is significant? I re-ran my analyses removing this correlation, and none of my fit indices or estimates changed
is the default in a regression with these 2 as independent variables, so that's why your fit indices don't change when you remove the term.
I now see your model more clearly - you have only 1 dependent variable, which is selfreg for the adolescent. I was responding as if you had several dependent variables such as one for the adolescent and one for the adult. Your model accounts for the intraclass correlation between the variables mom and adol, and Type=Complex accounts for the intraclass correlation for the dependent variable selfreg (adolescents clustered within family). So you are doing things right as far as I can tell.
Could you tell me how (or whether) the size of bivariate correlations among the lower-order units influence intra-class correlations? Let's say that I have couples as higher-order units, and individuals within these dyads as lower-level units. Does the intraclass correlation depend on what the correlation is between the members of the dyads on variable x? Or, does the intra-class correlation depend only on means and variances around it?
I've seen several people suggest that the intraclass correlation for a binary outcome can be calculated by using (pi^2/3)for the estimate of the level 1 variance. What I'm not clear on is whether the "pi" is just the constant 3.14... or whether this is a value that has to be calculated from the data (e.g. the probability of the outcome in the null model). I would greatly appreciate it if someone could clarify this for me.
I simply want to calculate ICC with three variables; 1, my outcome, 2, my ID for individuals and 3, my ID for schools. All of the examples I've looked at include other school level variables for the between command. How can I just find out how much variation I have in my outcome at the individual and school level, with ICC.
Thank you. However, my outcome is categorical. Is it still the case that MPlus does not calculate an ICC with a categorical outcome? If so, is it possible to get the coefficients that would be necessary for me to calculate the ICC?
Hi, I am about to set up a (single-level) SEM based on a clustered sample (kids in school classes) by using cluster=codecl; and ANALYSIS: TYPE=COMPLEX; In the paper, I would like to report the intraclass correlation coefficients of the outcome variables. Following the Mplus Discussion, I saw that it is possible to obtain the ICC directly in the Mplus output but I don't know how to do this. Unfortunately, there is no information on ICC in the Users Guide.
Emmanuel, The ICC is obtained through TYPE=TWOLEVEL since TYPE=COMPLEX only corrects for the standard errors. If your outcome(s) are continuous, you will get the ICC values directly in your output. /Amir
Hi, I have a data set with clustering (4 schools, and 45 classrooms). When I inspect the ICCs and accompanying design effects across classrooms, the highest design effect is around 1.9. But, the highest design effect across schools is 2.8, and the design effects for 4 out 10 model variables revealed values higher than 2. Meanwhile, I didnt measure any variable at the classroom and school level. I have several questions regarding this model: 1. Would Type=Complex provide sufficient correction against clustering in such a case? Which 2. If I should use Type=Complex, what should be the clustering variable? School, or classroom? 3. Is it possible to estimate scaled chi-square and robust SE for both school and classroom in a single model? 4. Is it possible to include school or classroom as level 2 variable and estimate variations in the outcomes across these clusters in MPlus?
Four schools is not enough for TYPE=COMPLEX or TYPE=TWOLEVEL. You need between 30-50 schools for this. I suggest creating three dummy variables that represent the four schools and using those in the analysis to control for non-independence of observations.
xusihua posted on Friday, February 26, 2010 - 6:53 pm
hello, how to comparing intraclass corelations within a single group?
In team research, generally team-level analysis is done by aggregating variables to group level, by showing high ICC (>.8). However, this type of analysis does not look at individual-level analysis and just conducts group-level analysis. For example, social cohesion and group performance will be studied at group level. team members will provide perceptions of group bonding and these will be aggregated (reference shift consensus model, Chan 1998) and analysis will be done at group-level.
I am viewing these relationships using Multilevel SEM, my ICC for cohesion is .34 (and design effect > 2)- as given by MPlus. I am using cohesion at both team and individual level. Now a regular team researcher would also ask me to show high-level of aggregation for cohesion at group level. Do you think this expectation is justifiable. Cause if I understand correctly, if ICC is too low (let us say .01) then this variable is essentially an individual-level variable; likewise if ICC is too high (let us say .9) then the variable is essentially a group-level variable; and in both cases we don't need a multilevel model.
I hope if you can tell me that in MSEM, do i need to show high level of aggregation (ICC>.8) for a variable that exists at both levels? <i> Many thanks.
I have repeated observations of individuals within families. I am doing CFA with Clustering on family and individual, and using TYPE=COMPLEX.
I wanted to get ICCs of the indicators for descriptive purposes. (I have ICC for each indicator computed outside of MPlus, but I want ICC under FIML handling of missing data.) TYPE=TWOLEVEL COMPLEX is needed for more than one cluster variable, and requires MLR which does not give any ICC.
Can I calculate these? Do I need to get between variances for family and subject from separate TWOLEVEL models? Is there some way to use the MLR scaling correction instead? Is it like a design effect?
When using Type=TWOLEVEL, I am informed that only 1 cluster variable is permitted and that the number of cluster variables (i.e., individual, family) is exceeded. The error message says to use TYPE=TWOLEVEL COMPLEX
"*** ERROR in VARIABLE command Two cluster variables are allowed for TYPE=TWOLEVEL COMPLEX. Only one cluster variable is allowed for TYPE=COMPLEX (single level). Limit on the number of cluster variables reached."
I took from this that you can have one variable with TWOLEVEL and one variable with COMPLEX, and can only get two cluster variables with TWOLEVEL COMPLEX.
Am I missing something or makeing a bad assumption?
Yes. As I gathered. I think we are on the same page now. I did not have a problem in the main factor model because I only clustered on individual, and did not model the variance by level, or desire between and within factors.
But I did want to model the levels in an auxiliary (empty) model to get the ICCs under FIML. (I have them from a standard package based on non-missing data). Or does the FIML preserve the observed ICCs? Maybe the effort is pointless.
So is the answer simply that there is no way to get what I want in MPlus, or is there some way by estimating between-family and between-individual variance in separate twolevel models to calculate the ICC after FIML, or is there some way to approximate it from the MLR scaling correction? For example divide the correction by the average cluster size or something? This is what I was originally asking.
Well, I will make one last try. I must be expressing things poorly. Apologies.
Since MPlus can only do 2-level and not 3-level (except when lowest level is wide), is there any way, hand calculation, trick, mushing together of information from various models ... etc that will allow me to figure out 3-level ICCs after FIML?
Your advice about TWOLEVEL BASIC and only 1 cluster variable, only gives 2-level ICC. I am after 3-level ICC.
You can not obtain ICC's for the third level in Mplus.
Anonymous posted on Tuesday, August 24, 2010 - 11:04 pm
Dear @Prof. Muthen
I'm running a multilevel SEM with random slopes. My model is not calculated ICC. Am I right to assume that it is not possible to calculate ICC by multilevel SEM with random slopes? How is ICC calculated in my model, if m-plus can calculate ICC for multilevel SEM with random slopes? Please help me.
My model is as follows;
usevariables ARE mbo1-mbo3 jc1-jc5 clus subTI1g subTI4g situum; CLUSTER = clus; WITHIN = mbo1-mbo3 jc1-jc5; BETWEEN = subTI1g subTI4g situum;
My question is whether "median cluster size" instead of "mean cluster size" can be used. I ask because in my data about 95% of cluster sizes are quite small (up to 10) but few of them are much larger (around 60 or 70). These few clusters distort the "mean" of cluster size a lot due to sensitivity of mean and thus inflate design effect. Would median cluster size be more appropriate in this case? Thank you.
The formula above is the DEFF for a mean when all cluster sizes are the same so average cluster size is actually cluster size. It is an approximation in other cases but it seems like a useful one. See the following article which is available on the website:
Muthén, B. & Satorra, A. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267-316.
Using the median may yield better information in your case.
I'm trying to calculate ICCs for a set of level 1 variables by modeling an empty two-level model using "TYPE IS TWOLEVEL BASIC". I calculated this in HLM as well (by dividing the level 2 variance by level 1 + level 2 variance) and found that some of the ICCs were similar numbers but different in magnitude by 2 decimal places. For example:
In such a syntax below in which our covariates (e.g., gender, ses1, ses2) and the latent factor indicators (ques1-ques30) are categorical variables; if we want to calculate the icc values should we indicate these variables in the VARIABLES command with "CATEGORICAL ARE"?
TITLE: Intraclass Correlation Coefficients DATA: FILE IS icc.dat; VARIABLE: NAMES ARE class gender gpa ses1 ses2 ques1-ques30; USEVARIABLES ARE gender gpa ques1-ques30 ses1 ses2; CLUSTER IS class; ANALYSIS: TYPE IS TWOLEVEL BASIC; ESTIMATOR IS WLS; OUTPUT: SAMPSTAT STANDARDIZED;
So 1. we do not place the covariates on the CATEGORICAL list? 2. we do not place ques1-ques30 on the CATEGORICAL list as they are independent observed variables for instance; although where they are categorical indicators of
You should run Type = Twolevel Basic to get icc's.
Whether your model gets significant between-level variances depends on the model and may or may not agree with the icc's. For instance if a 1-factor model fits well, you might get significant between-level factor variance even with small icc's.
Joe King posted on Thursday, November 17, 2011 - 11:05 pm
when running a zero inflated negative binomial, is it possible to get an intra-class correlation or some analog to it?
Good question. There is a between-level variance clearly. The issue is if there is a within-level variance to be used for the icc computation. That leads to the question if one can view negbin as having a continuous latent response variable formulation - like logistic regression, where it can be viewed as having a residual with variance pi-square/3. One formulation of negbin expresses it as a generalization of the Poisson by capturing heterogeneity by adding a residual to the log(mean) linear regression, where exp(residual) has a Gamma distribution. So if one can deduce that exp(residual) variance (presumable as a function of the negbin variance parameter), I guess there is a chance for this icc.
Which is a long way of saying I don't know.
Joe King posted on Friday, November 18, 2011 - 6:18 pm
well i think this is similar to what has been stated in a recent article , titled Repeatability for Gaussian and non-Gaussian data: a practical guide for biologists by Shinichi Nakagawa and Holger Schielzeth, Biological Reviews,85, 935956, (2010) in a zero inflated negative binomial they given an equation that looks at this, they called it repeatability but it seems like a similar concept of capturing the variability between groups.
so in your response you are referring to the Poisson-Gamma mixture where gamma is capturing the over dispersion that exists the Poisson is not? when MPlus runs the NB, is this how it calculates the dispersion parameter? thank you for your willingness to discuss this.
I think you are saying it right. Mplus uses the "negbin2" parameterization and estimates its dispersion parameter alpha, the details of which are given in Hilbe's negbin book (now in second edition).
I have not seen papers on this topic - I have never seen an icc with any count model. That of couse, doesn't mean that you can't estimate the magnitude of level 2 variance, that is, the numerator of the usual icc.
Joe King posted on Friday, November 18, 2011 - 6:30 pm
i have Hilbe's 2nd ed book, its very good and the NB2 is what i was thinking about. I havent seen it with any count models either but wanted to make sure i wasnt missing anything. Thank you Dr. Muthen.
Regarding the ICC reported in the output for a TWOLEVEL regression; the ICC value changes for the dependent variable depending on the predictors conditioned on (especially after adding variables at level 2).
I recognize that you have told us to use TWOLEVEL BASIC to obtain ICC values. But what do these other ICCs represent? They don't appear to be the ML estimates of the between residual variance relative to the total residual variance (but they are close).
Thank you, Laura
Antti Kärnä posted on Friday, December 09, 2011 - 9:44 am
Hello, I have been planning 2-level CFA with ordinal indicators. When I fit the model using the WLSMV estimator, I get the ICCs for the 12 items:
Linda- One additional question. When using Mplus version 6 TYPE=TWOLEVEL BASIC, is the ICC still calculated as shown in formula 203 (using the variance components determined with 197-202) in the Technical Appendix 10?
Calculating by hand using both MSb and MSw from an ANOVA (and a scaling factor) or using components from PROC CANDISC from SAS, I get fairly different estimates. For one of my variables, I get an ICC of .17 and Mplus gives me .23.
Would be glad to send the data if you want to test. Laura
Formula (203) applies for the MUML estimator. For the rest of the estimators ML/MLR/MLF the unrestricted H1 model is estimated to get the within and between variance.
Usually the ICC changes with the covariates due to this misspecification where a covariate is on the within list but actually it is not a within level variable because it hasn't been centered. You can use this command to fix this misspecifcations: centering=GROUPMEAN(x); for all x variables that are on the within= list.
When the sample is not large the ICC can also change when you add covariates due to the fact that the covariates carry more information - in particular information about the between level component which would be measured more precisely. In that case you should just use the estimates with the covariates included.
These iccs do look a bit small. But the real test is doing the 2-level model and seeing (1) if the results are different than what you get when ignoring clustering, and more importantly (2) if you get any significant between-level parameter estimates.
Tihomir- Thank you for the explanation. My data included the group mean of the variable in question on the USEVARIABLES list so including that in the unrestricted H1 model for the BASIC run resulted in an inappropriate ICC calculation. (Or at least inappropriate in terms of what I personally was expecting).
In Mplus there are two estimation of the Sigmas: one is based on the sample statistics and is reported as covariances (within and between) and the second is reported in the model results. These two estimations are not equivalent and ICC estimation is based on the first. Why is that?
We base the ICC's on the sample statistics because we want to see the intraclass correlations in the data.
Emil Coman posted on Wednesday, February 01, 2012 - 1:26 pm
I have tried to obtain the significance of the ICC using Bengt's above suggestion (Thursday, June 15, 2006 - 8:41 am), so I wrote: Usevariables are cidibtot ; cluster is educ; ANALYSIS: TYPE=TWOLEVEL BASIC; Model: %Within% cidibtot (vw1); %between% cidibtot (vb1); Model constraint: new(icc1 ); icc1 = vb1 / (vb11+vw1); BUT, there is no icc1 parameter listed in output. What am I messing up here? Thanks, Emil
Can you refer me to a source or an annotated output which explains how Mplus calculates the sample statistics in a TWOLEVEL analysis? At the moment, the most confusing thing to me is that the sample statistics of the dependent variable (e.g. variances, ICC) differ dependent on specifications for the independent variable (eg. centering, adding/deleting a predictor, using x and xw vs. decomposing x into a latent within and a latent between variable etc.). How do the sample statistics in a TWOLEVEL analysis relate to those obtained by TWOLEVEL BASIC?
I'm doing a twolevel analysis with pupils clustered in classes. Is it possible that the ICC is quite small (0.012, average cluster size = 19) but the explained variance on level 2 is 86 %? I just want to know if it is a contradiction or if it might be possible from a statistical point of view.
I experience that growth models using different approaches (multilevel or not, ML or Bayes) do not give consistent results. In particular, using two parallel growth models I now get a "significant" result with Bayesian (diffuse priors) for a cross-lagged path between Intercept and Growth factor at the individual level when using a multilevel model and correlated intercept/growth factors at the between level. Single-level analysis gives no "significant" path, and a multilevel model imposing a (questionable?) structural model at the between level also gives no significant path at the within level (nor at the between level). Multilevel modelling will often result in the opposite of what I get (in my case, I go from non-significant to significant coefficients when moving to multilevel).
ICCs in the current analysis are moderate , varying from less than .03 and up to over .07 (three-wave data). Average cluster size is 6.29.
I would normally tend to trust results from multilevel modeling more, but it appears to me that the results are not robust. Moreover, the design effect seems not particularly large?
Agnes Szabo posted on Tuesday, March 05, 2013 - 6:13 pm
Hello Linda and Bengt,
I am doing a CFA with a sample of 9000 students from 96 schools. The intraclass correlations are very small (lower than 0.01) and the values of DEFF are lower than 2. Do I need to use multilevel modelling with my data? (The multilevel structure is not of interest, I am just interested in the psychometric properties of the scale.)
It sounds like you would not need to take non-independence of observations into account in your case. You could run the model with and without TYPE=COMPLEX to see if there are any differences.
Melvin C Y posted on Wednesday, March 13, 2013 - 5:05 am
My data consist of student (n=1000), class (n=100) and school (n=25). On average I have 10 students per class and 4 classes per school. I am trying to obtain ICCs for a class level variable (teacher experience) nested in schools. I thought this should be quite simple but I'm confused about the cluster sizes that Mplus generates.
The syntax is below: . usevar=exp; cluster=school; analysis: type = basic twolevel;
The summary of data correctly displayed 25 clusters(schools). But the average cluster size was 40 which seemed to reflect the number of students rather than classes per school. ICC is decent at .14
In another run, I removed all student variables in my data, keeping only the school ID and the class level variable (exp). The average cluster size reported (per school) was 4, which is correct. But ICC is now .06.
Should a three level dataset be set up differently if I'm only interested in class and school level analysis? Or can I use the first procedure which perhaps account for student level structure (hence, more information?) even if it is not explicitly modelled.
Or, you can keep your data as they are and request threelevel analysis and add a student-level variable on your USEV list in addition to exp, where exp is declared a level-2 variable (see UG examples). This gives you icc's on two levels.
Melvin C Y posted on Wednesday, March 13, 2013 - 5:43 pm
Thanks Linda. As a follow up question, Hox (2010, p 34) wrote in his multilevel analysis book that there are two methods to compute 3 level ICCs at class and school level. L2/(L1+L2+L3) or (L2+L3)/(L1+L2+L3) The second formula assumes that two students from the same class must come from the same school. When school variance is large then values produced by both formula will be quite different. My questions are: 1) Assumung that I have student variable varying over class and school, which formula does Mplus use? 2) In my question above where the unit of analysis is class, what actually happens when I didn't remove all student cases? 3) Is creating a new dataset (without students) the only way to get Icc for class nested in school?
If you are running a three level analysis the ICCs that Mplus produces are L2/(L1+L2+L3) and L3/(L1+L2+L3). Note that there are two ICCs for each student level variable. If a variable is defined on the class level the ICC is computed the same way but L1=0.
deana desa posted on Monday, April 08, 2013 - 6:02 am
I would like to compare ICCs between two models which related to different specifications of between and within level weights.
Here is one part of the specification for model 1:
CLUSTER = idsc; WEIGHT = tw;
and here's for model 2: CLUSTER = idsc; WEIGHT = L1w; BWEIGHT = L2w; WTSCALE = CLUSTER; BWTSCALE = SAMPLE;
The ICCs derived from Mplus are as follows:
Model 1: Intraclass Correlation 0.417
Model 2: Intraclass Correlation 0.420
Can I know how can I verify these two values from mplus output? I tried to compute from the estimated values but I know that it's not right to do so.
I also saved the SIGB and SWM matrix files but how to read these file?
We dont have a way to compare these two estimates - however you can get a confidence interval for the ICC by specifying the unrestricted model in the model command and using a new parameter in model constraints to define the ICC as the new parameter.
By ICC(1,1) I mean a calculation based on a single rating to help determine how accurate a single rater may be, and by ICC(1,k) I mean a calculation based on the average of all raters within each group/cluster.
With the calculation described as BetweenVariance/(WithinVariance + BetweenVariance), I suspect the ICC given represents averages within each group.
Dear Drs. Muthén, in my multi-level model I have an observed DV, 2 observed IVs and two latent IVs.
To calculate ICC for DV and IVs I computed with TWOLEVEL BASIC one model with regression equation (including latent factor part as well as regression part "DV on IV1 IV2") and one model without regression equation (including only the latent factor part "IV1 by x1 x2 x3" on both levels).
Average sample size as well as ICCs vary between both models.
Which one should be reported - the ICC from the model with regression equation or from the model without regression?
Sorry for adding a follow-up question. I think your answer refers to ICC for observed variables. For this part of the model it helps me a lot.
However, I also would like to calculate the ICC for a latent factor and I think that I have to model this latent factor to get its' within variance and between variance for computation of its' ICC (variance within/total variance).
As I have to model the latent factor - should I model it with or without covariates?
May I know the standard ICC value for multilevel (two level) analysis in Mplus? What is the justification of ICC in within and between levels? Can you please recommend some references to show the cut-off value for ICC that analyse using Mplus?
I am testing a 1-2-model with a level 1 IV (employee work-family balance), a level 1 moderator (employee job autonomy), and a level 2 DV (academic reputation of the department). The average cluster size is 25 and the number of clusters is 44.
However, the ICCs of the IV and M are low (around .03). I understand the issues involving low ICCs in, for example, 2-1-models, but is it also problematic in 1-2-models? And if so, what could be a solution here? Aggregate the individual-level variables to Level 2?
I would not be concerned about that unless your specific focus is on estimating ICCs. Check the Hox multilevel book to see if he expresses concern about this - this book also discusses simulations that show performance at different cluster sizes etc.
Firstly thank you so much for your help so far! I have a quick question about ICC's I am hoping you could answer. I am wondering why MPLUS does not provide the ICC when producing a LGC model for varying times of observation (type = random)? Is there any way to calculate this from the output given by MPLUS? Please feel free to direct me to a resource that explains this if there is one.