Intraclass correlations PreviousNext
Mplus Discussion > Multilevel Data/Complex Sample >
Message/Author
 Anonymous posted on Thursday, September 23, 1999 - 4:17 pm
My intraclass correlations are very small. Do I really need to use multilevel modeling with my data?
 Linda K. Muthen posted on Friday, October 29, 1999 - 11:55 am
It is really not the size of the intraclass correlation that is the issue. It is the size of the design effect, which is a function of the intraclass correlation and the average cluster size. A design effect greater than 2 indicates that the clustering in the data needs to be taken into account during estimation. The design effect is approximately equal to

1 + (average cluster size - 1)*intraclass correlation

So if the average cluster size is 50, an intraclass correlation of .03 would yield a design effect of approximately 2.47.
 Joop Hox posted on Tuesday, November 16, 1999 - 1:15 am
Linda, do you have a source for your rule of thumb that a design effect > 2.0 is worth taking into account?
 Linda K. Muthen posted on Tuesday, November 16, 1999 - 7:59 am
My source is personal communication, Bengt Muthen. Seriously, however, Bengt has no reference for this rule of thumb besides his paper with Albert Satorra in Sociological Methodology in 1995. In Table 2, results are reported for a Monte Carlo study. Although design effects are not presented, the approximate design effects can be computed. It seems approximate design effects of less than two do not result in overly exaggerated rejection proportions at the 5 percent level using Method 1. For example, with an intraclass correlation of .05, the approximate design effects for cluster sizes of 7, 15, and 30 are 1.3, 1.7, and 2.45, respectively. As shown in the table the 5 percent reject proportions are satisfactory for 7 and 15 but not for 30. Table 3 gives corresponding results for standard errors.

It would be interesting to hear other people's experiences about which size of design effects are considered important.
 Judith Gonzalez-Calvo posted on Thursday, December 02, 1999 - 12:35 pm
I have an inquiry regarding the use of a longitudinal study (NLSY79) that employs a complex sample design (DEFF estimated at 1.5 to 1.3 at various time point). I am testing a model that has a continuous outcome variable, but categorical exogenous (indirect and direct effects) and "mediating" endogeneous variables that are categorical. The exogenous variables are time-bound, i.e., they occur prior to my outcome (net worth in 1996). Does Mplus handle data from complex sample designs and is SEM the best way to proceed?
 Linda K. Muthen posted on Thursday, December 02, 1999 - 12:51 pm
Mplus does handle data from some complex sample designs for continuous outcome variables. So if you have categorical mediating variables, this would be a problem. However, with your small DEFF's, perhaps you could estimate the model as if it is a random sample. Then Mplus could handle the categorical mediating variables. If needed, you could still use weights in the analysis.

Regarding the appropriateness of SEM for your problem, a growth model of the type you describe can be estimated using SEM. The latent variables are used to represent the growth factors. See the examples section of the website for some examples of growth models.
 SUNGWORN posted on Friday, April 07, 2000 - 9:33 pm
I try to study multilevel topic that was proposed by Muthen, but I do not understand the exact meaning of this equation...Sigma T = SigmaW + C*SigmaB.
1. why does he scale SigmaB with C and what is C*SigmaB represent?
2. How does this equation differ from .....Sigma T = SigmaW + SigmaB... in ANOVA ?
3. How does intraclass correlation from ANOVA (ICC = SSB/SST) differ form HLM program.?
 bmuthen@ucla.edu posted on Saturday, April 08, 2000 - 8:48 am
These quantities are discussed on page 299 of the Mplus User's Guide. In equation (156) it says that SigmaW + s*SigmaB is the expected value of SB, that is the expected value of the between covariance matrix in the sample. In this equation SigmaB needs to be scaled with the average cluster size s (for further details, see, e.g. Muthen, 1990). SigmaT = SigmaW + SigmaB as usual. The intraclass correlation is defined in equation (160) as the ratio of between and total variances; I would expect this is the same as in HLM.
 Anonymous posted on Tuesday, May 15, 2001 - 11:57 am
I am interested in looking at correlations between variables for data that have a multi-level structure (individuals within communities). From my rudimentary understanding of Mplus, I should be able to use Mplus to obtain correlations within communities and estimates of correlations between communities. If this assumption is correct, I have two questions:

1) How does Mplus handle dichotomous variables at the aggregate level? If these variables were coded 1/0, would Mplus sum these values at the aggregate (community) level or average these values?

2) How would I compute the degrees of freedom and test the significance of the correlations at each level (within and between)?

Thanks...
 bmuthen posted on Wednesday, May 16, 2001 - 8:30 am
Mplus multilevel modeling currently only handles continuous dependent variables. Dichotomous variables are, however, allowed if they are not dependent variables. Individual-level variables are essentially community-level means at the community level. Significance of correlations on either level can be computed using t ratios of parameter estimates to standard errors.
 Anonymous posted on Tuesday, August 20, 2002 - 2:23 pm
I'm confused about the relationship between the Mplus hierarchical SEM and the Mplus options for the analysis of complex sample survey data.

Doesn't hierarchical modeling already take into account design effects, or does it only adjust the parameter estimates but not the SEs of the estimates for clustered data ? I thought conventional HLM analyses inherently take clustering into account.

Thanks.
 Linda K. Muthen posted on Tuesday, August 20, 2002 - 2:50 pm
TYPE=COMPLEX adjusts the standard errors and chi-square fit statistic for clustering but not the parameter estimates. Yes, hierarchical modeling, ie, TYPE=TWOLEVEL, takes clustering into account. TYPE=TWOLEVEL models the clustering and therefore adjusts the parameter estimates, standard errors, and chi-square test statistic.
 Anonymous posted on Tuesday, August 20, 2002 - 4:26 pm
To follow up on your reply:

does this mean that the CLUSTER option is redundant in Mplus and does not need to specified when doing a hierarchical SEM analysis ?

Thanks again.
 Linda K. Muthen posted on Tuesday, August 20, 2002 - 5:06 pm
No, this is how Mplus knows which clusters individuals belong to so that multilevel modeling can be done.
 Anonymous posted on Thursday, September 30, 2004 - 6:44 am
I am doing a 2-level SEM, the latent factor INFOR is measured by five observed indicators (X1-X5). I find that once I change the the fixed value of observed indicators, (simply by switching the order position of measurement indicaotrs, ie. INFOR BY X1-X5, change to INFOR X2 X1 X3 X4 X5)

all model estimates seems indentical except for the variance of latent factor at both levels. As a result, the intraclass correlation of latent factor moves from 27% to 91%. So my quesiton is:

how to solve the problem? which indicator estimated should be fixed to one?
is the difference in ICC values of latent factor related to the scale of observed indicators?

Thanks a lot!
 Linda K. Muthen posted on Thursday, September 30, 2004 - 3:37 pm
I can understand that the variance of the factor will change depending on the item selected for setting the factor's metric but I don't understand why the intraclass correlation would change to dramatically. Can you please send the two outputs and the data to support@statmodel.com?
 Linda K. Muthen posted on Monday, October 04, 2004 - 4:03 pm
This invariance of the latent ICC can only be expected when the within loadings are held equal to the between loadings. Otherwise, different factors are considered. With loading invariance, you will find latent ICC invariance.
 Jim McMahon posted on Monday, October 11, 2004 - 11:42 am
The rule of thumb value of 2 or greater for the design effect to justify a multilevel model [Oct 29, 1999] is very interesting to me. I am working on multilevel models with dyadic data (2 subjects per group). The ICC would have to be 1.0 in order for the design effect to be 2, according to

1 + (average cluster size - 1)*intraclass correlation

This implies that one never needs to use HLM with dyadic data, although there is a growing literature on such models. Any thoughts on this? Thanks, Jim
 Bengt O. Muthen posted on Tuesday, October 12, 2004 - 5:04 pm
The rule of thumb is intended to give a flavor of when one would go wrong in terms of standard errors and chi-square when ignoring the multilevel structure of the data. Multilevel modeling can be done even when there are smaller design effects when the multilevel structure itself is of interest, as it clearly is with dyadic data.
 Tom Munk posted on Thursday, November 17, 2005 - 11:48 am
I have been trained to compute ICC as:
between variance/(between variance + within variance).

For example, I have a model with:
between variance = 140
within variance = 900
my icc calculation = .135

But MPLUS provides an ICC of .300.

How does MPLUS calculate ICC? How should it be interpreted?
 Linda K. Muthen posted on Thursday, November 17, 2005 - 2:17 pm
Yes, this is how we compute an intraclass correlation. See formula 203 Technical Appendix 10. I would have to see where you get your information to comment further. Please send your input, data, output, and license number to support@statmodel.com.
 Marco posted on Wednesday, December 14, 2005 - 8:18 am
Hello Linda, hello Bengt,

in the 1994-paper about MCA, it says that the ICC is a rough indication of between-variation (step 2 of the recommended 5 steps) and that the need of an multilevel analysis could be tested be restricting Sig-B = 0 in an MCA. How would I do the latter and is it worth the effort (compared to simply look at the ICC´s to determine the need of an multilevel strategy)?
Many thanks!
 bmuthen posted on Wednesday, December 14, 2005 - 8:25 am
You set Sig_B = 0 simply by saying (assuming 2 variables):

%between%

y1-y2@0;
y1 with y2@0;

Probably not worth the effort. Use the icc's instead.
 Florian Fiedler posted on Tuesday, January 24, 2006 - 6:47 am
I obtain invariant ICCs as well. When fitting an empty 2-group model on a data file I obtain two ICCs (one for each group), but when I specify the regression models on both levels, the dependent variable's ICCs change. Why is that? The regression model should refer to the level specific variances of the dependent variable without changing them. I thought the ICC of a dependent variable to be independent from explanatory variables.

When doing a kind of preface analysis checking the ICCs, I obtain higher values compared to those ICCs from the regression model. Which of those should I report?

Kind regards and thanks for your help.

Florian Fiedler.
 Linda K. Muthen posted on Tuesday, January 24, 2006 - 8:13 am
Pleas send the outputs from the models with and without covariates, your data, and license number to support@statmodel.com.
 Florian Fiedler posted on Tuesday, January 24, 2006 - 11:13 am
Ok, I just did. Hope this will help.
 Noona Kiuru posted on Thursday, June 15, 2006 - 2:27 am
I have a question concerning relations between intraclass correlations and statistical significance of between-level and within-level variance estimates in the multilevel context. I would like to test whether clusters become more homogeneous across time (two time points) or, alternatively, more differentiated from other clusters. I thought that this could be done by comparing estimates of between-variance in outcome variable between Time 1 and Time 2, on one hand, and estimates of within-variance between Time 1 and 2 on the other. This comparison could be made by constructing a model for the observed variables at Time 1 and Time 2 in which the between- and within –level variances of the observed outcome variables are set equal across time. However, one referee suggested that this is not a proper way to study whether the clusters become more homogeneous (or similar). He/she suggested that I should test statistical significance of the difference between the intraclass correlations of two time points (these are .18 and .34, respectively) instead. Do you have any ideas what would be the best way to study this issue (i.e., whether clusters become more homogeneous across time)? And if it should be done by comparing intracluster correlations how should I do this? Thanks
 Bengt O. Muthen posted on Thursday, June 15, 2006 - 8:41 am
The intraclass correlation is

VB/(VB + VW),

where VB is between-cluster and VW within-cluster variance. One way to handle the test of the icc being the same across time is to use parameter labels in Model and then use Model Constraint:

Model:

%Within%
y1-y2 (vw1-vw2);
%between%
y1-y2 (vb1-vb2);

Model constraint:

new(icc1 icc2 diff);
icc1=vb1/(vb1+vw1);
icc2=vb2/(vb2+vw2);
diff = icc2-icc1;

"diff" gives you the estimate and SE for the icc difference. For the model where you set the icc's equal, you simply add diif=0. You can also compare the loglikelihoods for these 2 models in a standard 1 df chi-square test.
 David Bard posted on Friday, August 11, 2006 - 12:04 pm
1) I'm aware of kappa coefficients for this, but I'm wondering if it's also appropriate to calculate an ICC for two-level regression models where the dependent variable is categorical. Can you simply divide the sum of the between level residual variances and twice their covariances (i.e., between variability) by the standardized variance of the probit (=1) or logit (=pi/sqt(3), right?) link functions, i.e., the total variance? I suspect I'm inaccurate, b/c the ICCs I've calculated in this fashion change substantially depending on the link function I use.
2) Again for a categorical DV two-level regression model, I've tried modeling the covariance between a random threshold and a random level-1 slope. I get an estimate, but the total between variance changes quite a bit when I include this term (variance doubles without the covariance modeled: .052 to .122). Does it make sense to have this term in the model in the categorical DV case, and should this change in the between-level variability be of concern?
 David Bard posted on Friday, August 11, 2006 - 2:16 pm
I've just realized I was using the standard deviation of the standardized logistic distribution when calculating the ICC described above. When I change it to a variance, pi^2/3, the "alleged" ICCs are much more similar across link functions. So I guess my question 1) now is simply whether this calculation is appropriate? Question 2) still stands as is.
 Bengt O. Muthen posted on Friday, August 11, 2006 - 5:20 pm
1) this is a somewhat reasonable approach I think, much like one can say it is somewhat reasonable to work with an r-square for a continuous latent response variable underlying a categorical dv.

2) yes, it makes sense to have this term included. If the variance of the slope is significant, I would include it. If that makes the results change a lot, perhaps that is because the likelihood is a lot better.
 maarten selfhout posted on Wednesday, October 11, 2006 - 6:56 am
Dear Bengt and Linda Muthen,

I have data on adolescents' and their best friends' delinquency on five waves and want to look at associations between the growth curve of the target adolescents and growth curve their best friends. So I want to look whether the slope of delinquency of the target adolescent is predicted by the intercept of deliqnuency of the best friend, and vice versa (and the intercept-intercept correlation and slope-slope correlation). What would be the best way to model such associations? I am aware that I am dealing with dyadic data which should be treated with a multilevel method, and therefore should look at intraclass correlations, but at the same time my aim is to look at the strength of the association between the intercepts, the slopes, and the intercept-slope correlations.
 Linda K. Muthen posted on Wednesday, October 11, 2006 - 10:04 am
What comes to mind when I read your message is the type of analysis suggested in the following paper:

Khoo, S.T. & Muthén, B. (2000). Longitudinal data on families: Growth modeling alternatives. Multivariate Applications in Substance use Research, J. Rose, L. Chassin, C. Presson & J. Sherman (eds.), Hillsdale, N.J.: Erlbaum, pp. 43-78. (#79)

You can request paper 79 from bmuthen@ucla.edu if you can't get the paper.
 Anup posted on Saturday, February 24, 2007 - 2:28 pm
What is the appropriate formula to calculate design effect for 3-level design? Are the formula presented above still applicable? Since there are 2 ICCs: level 2 and level 3 in such design, is there any adjustment needed in design effect formula 1 + (average cluster size - 1)*intraclass correlation ?
 Bengt O. Muthen posted on Saturday, February 24, 2007 - 3:24 pm
I don't know off hand what the formula is. It is probably covered in the classic Cochran book Sampling Techniques. Note that the formula you cite is for the special case of estimating means and with equal cluster sizes
 elisa posted on Tuesday, March 13, 2007 - 3:37 am
hello,
I'm carry on a multivel analysis, just an empty model for categorical variables.
How is possible to obtain the ICC with it's p value and the p value of the estimates (thresholds and variances)?
Thanks a lot
Regards,
Elisa
 Boliang Guo posted on Tuesday, March 13, 2007 - 3:56 am
ICC(logistic reg)=(Level2 Var)/(Level2 var+ (phi2/3))
look Snijders's book
 elisa posted on Wednesday, March 14, 2007 - 2:35 am
Thanks a lot,
but I would like to know if Mplus provides the ICC estimate "automatically", and if Mplus provides also the ICC p-value in order to have a direct test.
(I saw in another message posted here that Mplus provides an ICC, but I'm not able to find the command and I calculated it)

If Mplus does not provide the ICC p-value, how is possible to test its significance?

CATEGORICAL ARE responce ;
CLUSTER = group;
TYPE = twolevel random ;
ESTIMATOR = ml;
MODEL:
%WITHIN%;
%BETWEEN%
responce;

And, how is possible (if possible) to obtain automatically the p-value of the other estimates (thresholds and variances)?

Thanks
Elisa
 Boliang Guo posted on Wednesday, March 14, 2007 - 2:50 am
as I know, statistically significant level 2 variance is necessary for Multilevel analysis, ICC provide information of the degree of level 2 variance over total variance. I am using MLwiN, HLM, Mplus and WinBugs, only WinBugs can give uncertainty of ICC. well, I think you get no nore information from ICC than the significance of level 2 variance. I do not know the command for ICC, but I think you can use Model constraint to get ICC in Mplus.
maybe Prof Muthen can give more information on this topic.
 Linda K. Muthen posted on Saturday, March 17, 2007 - 9:24 am
Yes, you would use MODEL CONSTRAINT to create the ICC by dividing the estimated between variance by the sum of the estimated between and within variances. The program then gives a standard error for this parameter.
 Metin Ozdemir posted on Thursday, June 28, 2007 - 12:32 pm
I have a multilevel path model, that includes variables X1b (measured at the group level), Y1 and Y2 (measured at the individual level). The between group model suggest that X1b effects Y1b (average Y1 in groups) (X1b -- > Y1b).

The within group portion of the model suggests that Y1 effects Y2 (both at the individual level) (Y1 --> Y2).

I also want to test if Y1 mediate the relationship between X1b (group level) and Y2 (individual level)
(X1b --> Y1 --> Y2).

Is this possible in MPlus? If so, how can I develop a syntax, or is there any example that I can use?

Thanks,
 Mike Tobak posted on Thursday, June 28, 2007 - 1:10 pm
I wonder if Mplus can automatically produce intraclass correlations as well as between and within group covariance matrix without estimating any specific models.

I just want to check if my data is appropriate for multilevel analysis (I am afraid that the variance of certain variable is close to zero at the second level), and I don't need to test any model at this time.

I just tried a simple model and Mplus couldn't give the estimations due to nonpositive definite matrix. And I want to know if there is any option in Mplus to produce intracorrelations and between-/within-group covariance matrix alone.

Thanks.
 Linda K. Muthen posted on Thursday, June 28, 2007 - 3:30 pm
Re: Metin

See Example 9.3 in the Mplus User's Guide. I think this is very close to what you want.
 Linda K. Muthen posted on Thursday, June 28, 2007 - 3:32 pm
Yes, you can use TYPE=BASIC TWOLEVEL; to do this. You can use the SAMPLE and SIGB options of the SAVEDATA command to save the matrices. See the Mplus User's Guide for more information.
 Mike Tobak posted on Thursday, July 12, 2007 - 9:26 pm
I have a question about the between- and within- groups matrices. I think the scaled between-group matrix should be larger than pooled within group matrix, since S(between)=Cov(within)+c*Cov(between). However, the scaled between-group matrix calculated from my data (I programmed in Matlab using the given formula) is less than pooled within group matrix, producing negative (but each covariance/variance component is very small, close to zero) between groups matrix. Is there any possibility to have negative (but close to zero) estimated variance for between groups? It sounds weird, but I checked my code, and it works well for other data. Can I say that my data has no multilevel structure?
Thanks.
 Bengt O. Muthen posted on Friday, July 13, 2007 - 10:40 am
The formula you give is equation (199) in Appendix 10 on the Mplus web site. Note that this formula is for the expected value of SB, not for the sample SB, where the sample SB is defined in (197). Using this expected formula, the estimated SigmaB is given in (202) and yes it can obtain negative values indicating no between variation.
 Mike Tobak posted on Sunday, July 15, 2007 - 8:21 pm
Prof. Muthen, Thank you!

I used the correct formula as you said. I noticed that you mentioned (Muthen 1994, multilevel covariance analysis, sociological methods & research, vol.22, No.3, 376-398) that "the ML estimator of SigmaB is frequently not positive definite and might not even have positive variance estimates. This means that, in practice, we might have to resort to analyzing SB to get a notion of the SigmaB structure. Fortunately, experience shows that when it is possible to analyze both matrices, similar results are obtained." (on page 389). Does that mean that even if the ML estimator (given in (202)) has negative variance estimates, there could still be between variations and we can get similar results by analyzing SB?

I am a little confused here. According to your answers to my last question, there should be no between variations if the estimated SigmaB from (202) has negative values.

Thanks for your time and help!
 Bengt O. Muthen posted on Monday, July 16, 2007 - 11:29 am
We know too little of these situations to say anything with certainty. If the SigmaB estimate from (202) has negative values it is likely that there is no between variation in the population, but on the other hand there may be some. The quote from my paper indicates similar results when analyzing Sb and estimated SigmaB - "when it is possible to analyze both matrices". I would say that precludes the case of negative variances for estimated SigmaB. The best way to find out if there is between variation is to fit the full two-level model using FIML.
 Mike Tobak posted on Saturday, July 28, 2007 - 1:17 pm
Prof. Muthen, THANK YOU!

I wonder if we can test it by multilevel regression (with random slopes).

For instance, we fit a level-one model with variable y on two other level-one variables x1 and x2, and a level-two model of the random intercepts/slopes on a level-two variable W. We found that the intercepts/slopes from level-one model are indeed random effects (variance is significant). By doing this, can we say that the level-one units are not independent since intercepts/slopes are random effects, even thought we got negative (but close to 0) estimated SigmaB?

In fact, that is what happened to my data. The calculated intraclass correlations are close to 0, but the estimated SigmaB is negative (close to 0 though). Then I use multilevel regression (in HLM or SAS). if I specify an ANOVA model (without any exploratory variables at level-one and level-two), the intercept is not a random effect (hence agree with small intraclass correlations); but if I include some level-one exploratory variables, the random slopes of such exploratory variables are random effects (having significant variance tested at level-two model). Does this make sense to you? Small intraclass correlations but the slopes of exploratory variables are random across different groups?

Many thanks!!
 Linda K. Muthen posted on Sunday, July 29, 2007 - 10:36 am
Yes, ICC's can be close to zero while slopes are random. An ICC is similar to a random intercept.
 Mike Todd posted on Tuesday, September 25, 2007 - 3:43 pm
Hi all:

I am having trouble reconciling the variance component and the ICC values reported the output from my two-level linear regression model.

The values reported in the output are as follows:

ICC = 0.241
Within variance (Sw)= 2.769 (ML estimated sample value)
Between variance (Sb) = 1.140 (ML estimated sample value)

Sb/(Sb + Sw) = 0.292

Am I looking at the wrong parts of the output or am I missing a piece of the calculation?

Thanks for any guidance you can provide.

-Mike
 Linda K. Muthen posted on Tuesday, September 25, 2007 - 4:36 pm
We compute the intraclass correlations using the unbiased estimates obtained via SAMPSTAT or TYPE=BASIC not the consistent ML estimates from the model.
 Mike Todd posted on Tuesday, September 25, 2007 - 5:04 pm
Thanks for getting back to me so quickly, Linda.

It looks like the discrepancy was due to a misinterpretation of the output on my part.

The values I presented above were from a Monte Carlo run, where the sample stats from the first data replication were reported in the output. I (incorrectly) assumed that the accompanying ICC was based on these statistics.

When I took the raw data from the first replication and used them in a "non-Monte Carlo" two-level run, I obtained an ICC that is consistent with the sample statistics (i.e., ICC = 0.292).

Thanks again for your help.

-Mike
 Mike Todd posted on Wednesday, September 26, 2007 - 9:50 am
After sleeping on this, I realized I should have asked what exactly the ICC reported in the Monte Carlo output represents.

Is this described in any of the documentation or online?

Thanks again.
 Linda K. Muthen posted on Wednesday, September 26, 2007 - 11:13 am
They are the intraclass correlations for the first replication in Version 4.21. In earlier versions, they were for the last replication. Is this what you are asking?
 Mike Todd posted on Wednesday, September 26, 2007 - 11:56 am
Thanks, Linda. That's exactly what I was asking. The output I found confusing was generated under version 4.1. I just re-ran the Monte Carlo analysis under 4.21 and got the expected result. Thanks again!
 Tsz Hong, Chan posted on Friday, March 28, 2008 - 2:35 am
I just start using Mplus to see if some models require multilevel CFA, and am new to multilevel analysis. As I put in different numbers of dependent variables into a 2-level model, I find that the ICC of the variables are different. For instance, when I put in 5 DVs, the ICCs for these 5 DV are different when I put in one more DV. All my data do not contain missing value. May I ask the mechanism for this difference? Using other software does not seem to result in such difference.
 Linda K. Muthen posted on Friday, March 28, 2008 - 9:03 am
Please send the two outputs and your license number to support@statmodel.com.
 Sharon  posted on Saturday, March 29, 2008 - 11:02 am
I have a question relevant to a posting on May 15, 2001. Like that individual, I am interested in looking at correlations in a multilevel context. To do this, I have been examining the between and within correlation matrices using twolevel analyses. (a) Is it appropriate to interpret correlations involving two continuous variables in the within correlation matrix as the equivalent of standard rs, but corrected for the multi-level nature of the data (e.g., can I square them to get estimates of effect size)? (b) are there any special considerations in interpreting these correlations? (c) I'd like to report these correlations in articles I'm submitting; do you know of any examples where this has been done in the literature (or any problems people have had)?

Thanks for any guidance you can provide.
 Linda K. Muthen posted on Sunday, March 30, 2008 - 10:44 am
The within covariance is the covariance between the variables excluding the covariance due to clustering. In terms of correlations, you should decide if you want to calculate the correlations using total variances or within and between variances. Mplus uses within and between variances. I know of no special considerations in the interpretation of the correlations. I am not familiar with squaring a correlation to get effect size.
 Jeremy Flaherty posted on Saturday, May 10, 2008 - 12:48 pm
I have a simple question which I'm having trouble getting an answer to concerning the interpretation of ICCs. Take the following hypothetical example:

100 high schools are randomly sampled, from each of which 100 seniors are randomly sampled. SAT scores of the seniors are recorded. The ICC is found to be a statistically significant .01. Only 1 percent of the total variance in SAT scores is between schools.

Based on this small ICC, would it be wrong to interpret this as follows: "The school a student attends is not particularly relevant to the student's performance on the SAT."

thanks for any help.
 Linda K. Muthen posted on Sunday, May 11, 2008 - 10:51 am
That sounds reasonable.
 Lotta Tynkkynen posted on Monday, June 09, 2008 - 5:28 am
Hello!

How to test difference of ICCs between two groups (in this case gender)?

Lotta
 Linda K. Muthen posted on Monday, June 09, 2008 - 9:46 am
You can use MODEL CONSTRAINT. See the user's guide for more information.
 Daniel Alima posted on Tuesday, July 15, 2008 - 12:00 pm
Hi, my understanding of the intraclass correlation for multilevel analysis is that it is like the R-square in the linear regression (i.e., how much variance is explained by the between-cluster variance). However, if the dependent variable is count data, does intraclass have substantive meaning? And, can I obtain it from MPLUS analysis? Thanks!
 Linda K. Muthen posted on Wednesday, July 16, 2008 - 3:12 pm
Your understanding is correct. We don't currently know how to define a variance/residual variance for a count variable.
 Ramzi Mabsout posted on Tuesday, August 26, 2008 - 12:12 am
Hello

Am I right to assume that it is not possible to calculate intraclass correlations with (dependent) categorical variables only?

Thank you!
 Linda K. Muthen posted on Tuesday, August 26, 2008 - 9:23 am
See short course Topic 7A slides 44-49. The ICC for categorical outcomes is shown on slide 49.
 Ramzi Mabsout posted on Tuesday, August 26, 2008 - 10:43 am
Thank you Linda.

But it is still not clear to me whether I need to specify a model first to get these correlations?
 Linda K. Muthen posted on Tuesday, August 26, 2008 - 11:18 am
You can use TYPE=TWOLEVEL BASIC to obtain the information to compute the ICC's for categorical outcomes. We do not do this in Mplus.
 Boliang Guo posted on Wednesday, August 27, 2008 - 7:18 am
I remember there is a formula in Prof Muthen's Utrchet lecture. for random intercept only model, the ICC for 2-level logistic regression,
ICC(logistic reg)=(Level2 Var)/(Level2 var+ (phi2/3))
look Snijders's book, you pls check Mplus discussion for some answers
 Michael W Myers posted on Wednesday, August 27, 2008 - 3:28 pm
Hi,
I'm running a simple path analysis in which adolescent and parental negative emotional expression predict adolescent self-regulation. To account for non-independence of my data, I used TYPE=COMPLEX and clustered it by family (805 families).
In my model, there is a significant path between my exogenous variables of adolescent negative emotional expression and parental negative emotional expression. Since I've clustered my data by family, is this correlation the same as an interclass correlation? This is my first time using the TYPE=COMPLEX command, so I'm a little hesitant in interpreting my output.
thank you for your help!
 Bengt O. Muthen posted on Wednesday, August 27, 2008 - 3:46 pm
My first impression is that you account for non-independence twice - once is enough.

Type=Complex would treat family members as members of the cluster=family. If a family member is observed on p variables, there are p observed variables in the model.

If on the other hand you have a parent and an adolescent in each family and each is observed on p variables, there are 2*p variables in the model. When you say that parental expression is regressed on adolescent expression, this is the situation that comes to my mind. In this case, Type = Complex is superflous because the intra-family correlation between parent and adolescent is modeled as you say.

For related work in a growth model framework, see

Khoo, S.T. & Muthén, B. (2000). Longitudinal data on families: Growth modeling alternatives. Multivariate Applications in Substance use Research, J. Rose, L. Chassin, C. Presson & J. Sherman (eds.), Hillsdale, N.J.: Erlbaum, pp. 43-78.

at my UCLA web site (#79).
 Michael W Myers posted on Thursday, August 28, 2008 - 12:21 pm
Hi Bengt,
Thank you for the quick reply and reference suggestion! May I double-check that I understand your response? My model looked like this:
selfreg on mom adol;
mom with adol;
Since I already clustered my "mom" and "adol" predictors by family using the TYPE=COMPLEX, are you saying that allowing "mom" and "adol" to be correlated in my model is redundant? If that is true, should I assign an meaning to that correlation in my model, given that it is significant? I re-ran my analyses removing this correlation, and none of my fit indices or estimates changed
 Bengt O. Muthen posted on Friday, August 29, 2008 - 7:58 am
Your statement

mom with adol;

is the default in a regression with these 2 as independent variables, so that's why your fit indices don't change when you remove the term.

I now see your model more clearly - you have only 1 dependent variable, which is selfreg for the adolescent. I was responding as if you had several dependent variables such as one for the adolescent and one for the adult. Your model accounts for the intraclass correlation between the variables mom and adol, and Type=Complex accounts for the intraclass correlation for the dependent variable selfreg (adolescents clustered within family). So you are doing things right as far as I can tell.
 Kätlin Peets posted on Tuesday, October 28, 2008 - 6:35 am
Hello,

Could you tell me how (or whether) the size of bivariate correlations among the lower-order units influence intra-class correlations?
Let's say that I have couples as higher-order units, and individuals within these dyads as lower-level units. Does the intraclass correlation depend on what the correlation is between the members of the dyads on variable x? Or, does the intra-class correlation depend only on means and variances around it?

Thank you!
 Linda K. Muthen posted on Tuesday, October 28, 2008 - 7:57 am
The intraclass correlation is computed using variances. It is affected by other model parameters only indirectly if model misspecification causes the variances to be misestimated.
 Brian Johnson posted on Friday, May 01, 2009 - 10:18 am
I've seen several people suggest that the intraclass correlation for a binary outcome can be calculated by using (pi^2/3)for the estimate of the level 1 variance. What I'm not clear on is whether the "pi" is just the constant 3.14... or whether this is a value that has to be calculated from the data (e.g. the probability of the outcome in the null model). I would greatly appreciate it if someone could clarify this for me.
 Linda K. Muthen posted on Friday, May 01, 2009 - 10:42 am
Pi is the number 3.14...
 Brian Johnson posted on Friday, May 01, 2009 - 11:06 am
Thank you so much for your quick response.
 Sherri Bisset posted on Friday, September 04, 2009 - 9:08 am
I simply want to calculate ICC with three variables; 1, my outcome, 2, my ID for individuals and 3, my ID for schools. All of the examples I've looked at include other school level variables for the between command.
How can I just find out how much variation I have in my outcome at the individual and school level, with ICC.
 Bengt O. Muthen posted on Friday, September 04, 2009 - 12:26 pm
Use Type = Basic;

So no MODEL command.
 Sherri Bisset posted on Tuesday, September 08, 2009 - 9:48 am
Thank you.
However, my outcome is categorical.
Is it still the case that MPlus does not calculate an ICC with a categorical outcome? If so, is it possible to get the coefficients that would be necessary for me to calculate the ICC?
 Linda K. Muthen posted on Tuesday, September 08, 2009 - 10:52 am
Mplus does not give ICC's for categorical outcomes. See Slide 66 in the short course handout for Topic 7 for the formula.
 Michael Spaeth posted on Tuesday, September 22, 2009 - 10:13 am
Hi! I wanted to calculate ICCs for the binary measures of a two-part LGM. However, mplus provides no variances for the categorical outcomes. I guess this is because I'm using MLR!?
 Linda K. Muthen posted on Tuesday, September 22, 2009 - 10:34 am
See the Topic 7 course handout starting with slide 60 looking at slide 66 for the ICC formula. This can be done only with maximum likelihood.
 Emmanuel Kuntsche posted on Tuesday, January 26, 2010 - 4:15 am
Hi, I am about to set up a (single-level) SEM based on a clustered sample (kids in school classes) by using cluster=codecl; and ANALYSIS: TYPE=COMPLEX;
In the paper, I would like to report the intraclass correlation coefficients of the outcome variables. Following the Mplus Discussion, I saw that it is possible to obtain the ICC directly in the Mplus output but I don't know how to do this. Unfortunately, there is no information on ICC in the Users Guide.
 Amir Sariaslan posted on Tuesday, January 26, 2010 - 7:38 am
Emmanuel,
The ICC is obtained through TYPE=TWOLEVEL since TYPE=COMPLEX only corrects for the standard errors. If your outcome(s) are continuous, you will get the ICC values directly in your output.
/Amir
 Linda K. Muthen posted on Tuesday, January 26, 2010 - 7:51 am
Intraclass correlations are given with TYPE=TWOLEVEL. You can do a TYPE=TWOLEVEL BASIC to get them.
 Metin Ozdemir posted on Tuesday, February 02, 2010 - 5:45 am
Hi,
I have a data set with clustering (4 schools, and 45 classrooms). When I inspect the ICCs and accompanying design effects across classrooms, the highest design effect is around 1.9. But, the highest design effect across schools is 2.8, and the design effects for 4 out 10 model variables revealed values higher than 2. Meanwhile, I didn’t measure any variable at the classroom and school level.
I have several questions regarding this model:
1. Would Type=Complex provide sufficient correction against clustering in such a case? Which
2. If I should use Type=Complex, what should be the clustering variable? School, or classroom?
3. Is it possible to estimate scaled chi-square and robust SE for both school and classroom in a single model?
4. Is it possible to include school or classroom as level 2 variable and estimate variations in the outcomes across these clusters in MPlus?
 Linda K. Muthen posted on Tuesday, February 02, 2010 - 10:59 am
Four schools is not enough for TYPE=COMPLEX or TYPE=TWOLEVEL. You need between 30-50 schools for this. I suggest creating three dummy variables that represent the four schools and using those in the analysis to control for non-independence of observations.
 xusihua posted on Friday, February 26, 2010 - 6:53 pm
hello, how to comparing intraclass corelations within a single group?
 Bengt O. Muthen posted on Saturday, February 27, 2010 - 6:20 am
If you mean testing the equality of iccs across variables you can do that by defining the icc's in terms of model parameters in Model Constraint and using Model Test to test their equality.
 M Hamd posted on Friday, April 09, 2010 - 4:28 pm
In team research, generally team-level analysis is done by aggregating variables to group level, by showing high ICC (>.8). However, this type of analysis does not look at individual-level analysis and just conducts group-level analysis. For example, social cohesion and group performance will be studied at group level. team members will provide perceptions of group bonding and these will be aggregated (reference shift consensus model, Chan 1998) and analysis will be done at group-level.

I am viewing these relationships using Multilevel SEM, my ICC for cohesion is .34 (and design effect > 2)- as given by MPlus. I am using cohesion at both team and individual level. Now a regular team researcher would also ask me to show high-level of aggregation for cohesion at group level. Do you think this expectation is justifiable. Cause if I understand correctly, if ICC is too low (let us say .01) then this variable is essentially an individual-level variable; likewise if ICC is too high (let us say .9) then the variable is essentially a group-level variable; and in both cases we don't need a multilevel model.

I hope if you can tell me that in MSEM, do i need to show high level of aggregation (ICC>.8) for a variable that exists at both levels?
<i>
Many thanks.
 Linda K. Muthen posted on Saturday, April 10, 2010 - 8:16 am
It is not necessary for the ICC to be high to model a variable on both levels. Typically ICC's are fairly low, around .1 or .2.
 M Hamd posted on Saturday, April 10, 2010 - 10:27 am
Thank you Dr. Linda.
 Daniel E Bontempo posted on Tuesday, April 27, 2010 - 10:15 am
I have repeated observations of individuals within families. I am doing CFA with Clustering on family and individual, and using TYPE=COMPLEX.

I wanted to get ICCs of the indicators for descriptive purposes. (I have ICC for each indicator computed outside of MPlus, but I want ICC under FIML handling of missing data.) TYPE=TWOLEVEL COMPLEX is needed for more than one cluster variable, and requires MLR which does not give any ICC.

Can I calculate these? Do I need to get between variances for family and subject from separate TWOLEVEL models? Is there some way to use the MLR scaling correction instead? Is it like a design effect?
 Linda K. Muthen posted on Tuesday, April 27, 2010 - 3:43 pm
You can get the ICC's using TYPE=TWOLEVEL BASIC; You don't need to include COMPLEX for the ICC's.
 Daniel E Bontempo posted on Thursday, April 29, 2010 - 11:36 am
When using Type=TWOLEVEL, I am informed that only 1 cluster variable is permitted and that the number of cluster variables (i.e., individual, family) is exceeded. The error message says to use TYPE=TWOLEVEL COMPLEX

"*** ERROR in VARIABLE command
Two cluster variables are allowed for TYPE=TWOLEVEL COMPLEX. Only one
cluster variable is allowed for TYPE=COMPLEX (single level). Limit on
the number of cluster variables reached."


I took from this that you can have one variable with TWOLEVEL and one variable with COMPLEX, and can only get two cluster variables with TWOLEVEL COMPLEX.

Am I missing something or makeing a bad assumption?
 Linda K. Muthen posted on Thursday, April 29, 2010 - 1:54 pm
Only one cluster variable can be used for TOWOLEVEL or COMPLEX. Only when they are used together can two cluster variables be used.
 Daniel E Bontempo posted on Thursday, April 29, 2010 - 3:14 pm
Yes. As I gathered. I think we are on the same page now. I did not have a problem in the main factor model because I only clustered on individual, and did not model the variance by level, or desire between and within factors.

But I did want to model the levels in an auxiliary (empty) model to get the ICCs under FIML. (I have them from a standard package based on non-missing data). Or does the FIML preserve the observed ICCs? Maybe the effort is pointless.

So is the answer simply that there is no way to get what I want in MPlus, or is there some way by estimating between-family and between-individual variance in separate twolevel models to calculate the ICC after FIML, or is there some way to approximate it from the MLR scaling correction? For example divide the correction by the average cluster size or something? This is what I was originally asking.
 Linda K. Muthen posted on Friday, April 30, 2010 - 8:26 am
If you use TYPE=TWOLEVEL BASIC with the cluster variable that goes with TWOLEVEL, you will obtain ICC's computed with FIML.
 Daniel E Bontempo posted on Friday, April 30, 2010 - 9:59 am
Well, I will make one last try. I must be expressing things poorly. Apologies.

Since MPlus can only do 2-level and not 3-level (except when lowest level is wide), is there any way, hand calculation, trick, mushing together of information from various models ... etc that will allow me to figure out 3-level ICCs after FIML?

Your advice about TWOLEVEL BASIC and only 1 cluster variable, only gives 2-level ICC. I am after 3-level ICC.
 Linda K. Muthen posted on Saturday, May 01, 2010 - 4:12 pm
You can not obtain ICC's for the third level in Mplus.
 Anonymous posted on Tuesday, August 24, 2010 - 11:04 pm
Dear @Prof. Muthen

I'm running a multilevel SEM with random slopes. My model is not calculated ICC.
Am I right to assume that it is not possible to calculate ICC by multilevel SEM with random slopes? How is ICC calculated in my model, if m-plus can calculate ICC for multilevel SEM with random slopes? Please help me.

My model is as follows;

usevariables ARE mbo1-mbo3 jc1-jc5
clus subTI1g subTI4g situum;
CLUSTER = clus;
WITHIN = mbo1-mbo3 jc1-jc5;
BETWEEN = subTI1g subTI4g situum;

ANALYSIS:
TYPE = twolevel random;
algorithm = integration;
estimator = ML;

MODEL:
%WITHIN%
fwmbo by mbo3@1
mbo2(1)
mbo1(2);
fwjc by jc5@1
jc4(3)
jc3(4)
jc2(5)
jc1(6);
s1 | fwjc ON fwmbo;

%BETWEEN%
s1 ON subTI1g subTI4g situum;
subti4g WITH subti1g;
subti1g on situum;
subti4g on situum;
OUTPUT: tech1 tech8;
 Linda K. Muthen posted on Wednesday, August 25, 2010 - 3:43 pm
We do not compute ICC's with TYPE=RANDOM. It is not clear how this would be done.
 Anonymous posted on Wednesday, August 25, 2010 - 10:01 pm
Thank you so much for your quick response.
 Jan Stochl posted on Friday, August 27, 2010 - 7:05 am
Hello, I would like to ask again about the design effect size computation. As mentioned in this thread, the formula is

1 + (average cluster size - 1)*intraclass correlation

My question is whether "median cluster size" instead of "mean cluster size" can be used. I ask because in my data about 95% of cluster sizes are quite small (up to 10) but few of them are much larger (around 60 or 70). These few clusters distort the "mean" of cluster size a lot due to sensitivity of mean and thus inflate design effect. Would median cluster size be more appropriate in this case?
Thank you.
 Linda K. Muthen posted on Saturday, August 28, 2010 - 7:59 am
The formula above is the DEFF for a mean when all cluster sizes are the same so average cluster size is actually cluster size. It is an approximation in other cases but it seems like a useful one. See the following article which is available on the website:

Muthén, B. & Satorra, A. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267-316.

Using the median may yield better information in your case.
 Janna Fuccillo posted on Wednesday, October 20, 2010 - 2:46 pm
Hello,

I'm trying to calculate ICCs for a set of level 1 variables by modeling an empty two-level model using "TYPE IS TWOLEVEL BASIC". I calculated this in HLM as well (by dividing the level 2 variance by level 1 + level 2 variance) and found that some of the ICCs were similar numbers but different in magnitude by 2 decimal places. For example:

ICCs calculated from HLM --> ICCs in MPlus output

0.00045 --> 0.041
0.00029 --> 0.029
0.00031 --> 0.031
0.00073 --> 0.053

Other variables with larger ICCs were more similar across the 2 methods:

0.322 --> 0.329
0.463 --> 0.453

Since MPlus reports only 3 significant figures for ICCs, is it possible that numbers smaller than .001 are reported in a larger scale? Is there any way to change the way this is reported?

Any insight into this would be greatly appreciated!
 Linda K. Muthen posted on Wednesday, October 20, 2010 - 3:10 pm
Please send the HLM and Mplus outputs and your license number to support@statmodel.com.
 Utkun Ozdil posted on Monday, March 07, 2011 - 2:03 am
In such a syntax below in which our covariates (e.g., gender, ses1, ses2) and the latent factor indicators (ques1-ques30) are categorical variables; if we want to calculate the icc values should we indicate these variables in the VARIABLES command with "CATEGORICAL ARE"?

TITLE: Intraclass Correlation Coefficients
DATA: FILE IS icc.dat;
VARIABLE: NAMES ARE class gender gpa
ses1 ses2 ques1-ques30;
USEVARIABLES ARE gender gpa
ques1-ques30 ses1 ses2;
CLUSTER IS class;
ANALYSIS: TYPE IS TWOLEVEL BASIC;
ESTIMATOR IS WLS;
OUTPUT: SAMPSTAT STANDARDIZED;

Thanks...
 Linda K. Muthen posted on Monday, March 07, 2011 - 6:58 am
Only dependent variables should be placed on the CATEGORICAL list.
 Utkun Ozdil posted on Monday, March 07, 2011 - 7:58 am
So
1. we do not place the covariates on the CATEGORICAL list?
2. we do not place ques1-ques30 on the CATEGORICAL list as they are independent observed variables for instance; although where they are categorical indicators of

"factor1 BY ques1-ques30"?

Thanks...
 Linda K. Muthen posted on Monday, March 07, 2011 - 8:09 am
You do not place covariates on the CATEGORICAL list. In a factor model, the factor indicators, quest1-ques30, are dependent variables so they should be placed on the CATEGORICAL list.
 Anne Chan  posted on Thursday, April 07, 2011 - 6:45 pm
Hello! I run an intercept-only model to get the ICC. The ICC looks ok (0.1, average cluster size: 49). But the between level variances is non-significant.

Does the significance of the between-level variances of the intercept-only model relate to the significance of the ICC?


Thanks!
 Bengt O. Muthen posted on Friday, April 08, 2011 - 5:23 pm
You should run Type = Twolevel Basic to get icc's.

Whether your model gets significant between-level variances depends on the model and may or may not agree with the icc's. For instance if a 1-factor model fits well, you might get significant between-level factor variance even with small icc's.
 Joe King posted on Thursday, November 17, 2011 - 11:05 pm
when running a zero inflated negative binomial, is it possible to get an intra-class correlation or some analog to it?
 Bengt O. Muthen posted on Friday, November 18, 2011 - 6:02 pm
Good question. There is a between-level variance clearly. The issue is if there is a within-level variance to be used for the icc computation. That leads to the question if one can view negbin as having a continuous latent response variable formulation - like logistic regression, where it can be viewed as having a residual with variance pi-square/3. One formulation of negbin expresses it as a generalization of the Poisson by capturing heterogeneity by adding a residual to the log(mean) linear regression, where exp(residual) has a Gamma distribution. So if one can deduce that exp(residual) variance (presumable as a function of the negbin variance parameter), I guess there is a chance for this icc.

Which is a long way of saying I don't know.
 Joe King posted on Friday, November 18, 2011 - 6:18 pm
well i think this is similar to what has been stated in a recent article , titled Repeatability for Gaussian and non-Gaussian data: a practical guide for biologists by Shinichi Nakagawa and Holger Schielzeth, Biological Reviews,85, 935–956, (2010) in a zero inflated negative binomial they given an equation that looks at this, they called it repeatability but it seems like a similar concept of capturing the variability between groups.

so in your response you are referring to the Poisson-Gamma mixture where gamma is capturing the over dispersion that exists the Poisson is not? when MPlus runs the NB, is this how it calculates the dispersion parameter? thank you for your willingness to discuss this.
 Bengt O. Muthen posted on Friday, November 18, 2011 - 6:26 pm
I think you are saying it right. Mplus uses the "negbin2" parameterization and estimates its dispersion parameter alpha, the details of which are given in Hilbe's negbin book (now in second edition).

I have not seen papers on this topic - I have never seen an icc with any count model. That of couse, doesn't mean that you can't estimate the magnitude of level 2 variance, that is, the numerator of the usual icc.
 Joe King posted on Friday, November 18, 2011 - 6:30 pm
i have Hilbe's 2nd ed book, its very good and the NB2 is what i was thinking about. I havent seen it with any count models either but wanted to make sure i wasnt missing anything. Thank you Dr. Muthen.
 Laura Stapleton posted on Friday, December 09, 2011 - 7:42 am
Linda et al.-

Regarding the ICC reported in the output for a TWOLEVEL regression; the ICC value changes for the dependent variable depending on the predictors conditioned on (especially after adding variables at level 2).

I recognize that you have told us to use TWOLEVEL BASIC to obtain ICC values. But what do these other ICCs represent? They don't appear to be the ML estimates of the between residual variance relative to the total residual variance (but they are close).

Thank you,
Laura
 Antti Kärnä posted on Friday, December 09, 2011 - 9:44 am
Hello,
I have been planning 2-level CFA with ordinal indicators. When I fit the model using the WLSMV estimator, I get the ICCs for the 12 items:

U1 0.036 U2 0.012 U3 0.000
U4 0.020 U5 0.039 U6 0.023
U7 0.009 U8 0.007 U9 0.018
U10 0.025 U11 0.026 U12 0.029

Are these ICCs too small as such to justify a between-level model? Thanks in advance!
 Laura Stapleton posted on Friday, December 09, 2011 - 1:16 pm
Linda-
One additional question. When using Mplus version 6 TYPE=TWOLEVEL BASIC, is the ICC still calculated as shown in formula 203 (using the variance components determined with 197-202) in the Technical Appendix 10?

Calculating by hand using both MSb and MSw from an ANOVA (and a scaling factor) or using components from PROC CANDISC from SAS, I get fairly different estimates. For one of my variables, I get an ICC of .17 and Mplus gives me .23.

Would be glad to send the data if you want to test.
Laura
 Tihomir Asparouhov posted on Friday, December 09, 2011 - 2:22 pm
Formula (203) applies for the MUML estimator. For the rest of the estimators ML/MLR/MLF the unrestricted H1 model is estimated to get the within and between variance.

Usually the ICC changes with the covariates due to this misspecification where a covariate is on the within list but actually it is not a within level variable because it hasn't been centered. You can use this command to fix this misspecifcations:
centering=GROUPMEAN(x);
for all x variables that are on the within= list.

When the sample is not large the ICC can also change when you add covariates due to the fact that the covariates carry more information - in particular information about the between level component which would be measured more precisely. In that case you should just use the estimates with the covariates included.

If this doesn't help send the example to support@statmodel.com.
 Bengt O. Muthen posted on Saturday, December 10, 2011 - 6:51 pm
Answer to Antti of December 9.

These iccs do look a bit small. But the real test is doing the 2-level model and seeing (1) if the results are different than what you get when ignoring clustering, and more importantly (2) if you get any significant between-level parameter estimates.
 Laura Stapleton posted on Tuesday, December 13, 2011 - 7:05 am
Tihomir-
Thank you for the explanation. My data included the group mean of the variable in question on the USEVARIABLES list so including that in the unrestricted H1 model for the BASIC run resulted in an inappropriate ICC calculation. (Or at least inappropriate in terms of what I personally was expecting).

Thanks,
Laura
 Patricio Cumsille posted on Wednesday, December 21, 2011 - 6:21 am
In Mplus there are two estimation of the Sigmas: one is based on the sample statistics and is reported as covariances (within and between) and the second is reported in the model results. These two estimations are not equivalent and ICC estimation is based on the first. Why is that?
 Linda K. Muthen posted on Wednesday, December 21, 2011 - 11:28 am
We base the ICC's on the sample statistics because we want to see the intraclass correlations in the data.
 Emil Coman posted on Wednesday, February 01, 2012 - 1:26 pm
I have tried to obtain the significance of the ICC using Bengt's above suggestion (Thursday, June 15, 2006 - 8:41 am), so I wrote:
Usevariables are cidibtot ;
cluster is educ;
ANALYSIS: TYPE=TWOLEVEL BASIC;
Model:
%Within%
cidibtot (vw1);
%between%
cidibtot (vb1);
Model constraint:
new(icc1 );
icc1 = vb1 / (vb11+vw1);
BUT, there is no icc1 parameter listed in output. What am I messing up here? Thanks, Emil
 Linda K. Muthen posted on Wednesday, February 01, 2012 - 2:18 pm
Please send the output and your license number to support@statmodel.com.
 Katrin Lintorf posted on Monday, April 30, 2012 - 1:34 am
Can you refer me to a source or an annotated output which explains how Mplus calculates the sample statistics in a TWOLEVEL analysis? At the moment, the most confusing thing to me is that the sample statistics of the dependent variable (e.g. variances, ICC) differ dependent on specifications for the independent variable (eg. centering, adding/deleting a predictor, using x and xw vs. decomposing x into a latent within and a latent between variable etc.). How do the sample statistics in a TWOLEVEL analysis relate to those obtained by TWOLEVEL BASIC?

Thank you
-Katrin
 Linda K. Muthen posted on Monday, April 30, 2012 - 8:41 am
Please send two outputs that illustrate this issue and your license number to support@statmodel.com.
 Martin Geisler posted on Monday, August 06, 2012 - 10:48 am
Hello,

I'm doing a twolevel analysis with pupils clustered in classes.
Is it possible that the ICC is quite small (0.012, average cluster size = 19) but the explained variance on level 2 is 86 %?
I just want to know if it is a contradiction or if it might be possible from a statistical point of view.

Thank you so much!
 Linda K. Muthen posted on Monday, August 06, 2012 - 1:40 pm
I think it is common to see large R-squares on between. Don't forget the 86% is explaining only the between-level variance which may be rather small.
 Christopher Bratt posted on Thursday, November 08, 2012 - 2:59 am
I experience that growth models using different approaches (multilevel or not, ML or Bayes) do not give consistent results. In particular, using two parallel growth models I now get a "significant" result with Bayesian (diffuse priors) for a cross-lagged path between Intercept and Growth factor at the individual level when using a multilevel model and correlated intercept/growth factors at the between level. Single-level analysis gives no "significant" path, and a multilevel model imposing a (questionable?) structural model at the between level also gives no significant path at the within level (nor at the between level). Multilevel modelling will often result in the opposite of what I get (in my case, I go from non-significant to significant coefficients when moving to multilevel).

ICCs in the current analysis are moderate , varying from less than .03 and up to over .07 (three-wave data). Average cluster size is 6.29.

I would normally tend to trust results from multilevel modeling more, but it appears to me that the results are not robust. Moreover, the design effect seems not particularly large?
 Linda K. Muthen posted on Thursday, November 08, 2012 - 9:39 am
Please send two key outputs that show the differences and your license number to support@statmodel.com.
 Agnes Szabo posted on Tuesday, March 05, 2013 - 6:13 pm
Hello Linda and Bengt,

I am doing a CFA with a sample of 9000 students from 96 schools. The intraclass correlations are very small (lower than 0.01) and the values of DEFF are lower than 2. Do I need to use multilevel modelling with my data? (The multilevel structure is not of interest, I am just interested in the psychometric properties of the scale.)

Thank you very much,
Agnes
 Linda K. Muthen posted on Wednesday, March 06, 2013 - 6:59 am
It sounds like you would not need to take non-independence of observations into account in your case. You could run the model with and without TYPE=COMPLEX to see if there are any differences.
 Melvin C Y posted on Wednesday, March 13, 2013 - 5:05 am
My data consist of student (n=1000), class (n=100) and school (n=25). On average I have 10 students per class and 4 classes per school. I am trying to obtain ICCs for a class level variable (teacher experience) nested in schools. I thought this should be quite simple but I'm confused about the cluster sizes that Mplus generates.

The syntax is below:
.
usevar=exp;
cluster=school;
analysis: type = basic twolevel;

The summary of data correctly displayed 25 clusters(schools). But the average cluster size was 40 which seemed to reflect the number of students rather than classes per school. ICC is decent at .14

In another run, I removed all student variables in my data, keeping only the school ID and the class level variable (exp). The average cluster size reported (per school) was 4, which is correct. But ICC is now .06.

Should a three level dataset be set up differently if I'm only interested in class and school level analysis? Or can I use the first procedure which perhaps account for student level structure (hence, more information?) even if it is not explicitly modelled.


Thank you.
Melvin
 Linda K. Muthen posted on Wednesday, March 13, 2013 - 12:40 pm
You would need to create a data set where the unit of analysis is class. I think this is what you did when you got an ICC of .06.
 Bengt O. Muthen posted on Wednesday, March 13, 2013 - 5:06 pm
Or, you can keep your data as they are and request threelevel analysis and add a student-level variable on your USEV list in addition to exp, where exp is declared a level-2 variable (see UG examples). This gives you icc's on two levels.
 Melvin C Y posted on Wednesday, March 13, 2013 - 5:43 pm
Thanks Linda. As a follow up question, Hox (2010, p 34) wrote in his multilevel analysis book that there are two methods to compute 3 level ICCs at class and school level. L2/(L1+L2+L3) or (L2+L3)/(L1+L2+L3)
The second formula assumes that two students from the same class must come from the same school. When school variance is large then values produced by both formula will be quite different. My questions are:
1) Assumung that I have student variable varying over class and school, which formula does Mplus use?
2) In my question above where the unit of analysis is class, what actually happens when I didn't remove all student cases?
3) Is creating a new dataset (without students) the only way to get Icc for class nested in school?

Thank you.
 Tihomir Asparouhov posted on Thursday, March 14, 2013 - 9:05 am
If you are running a three level analysis the ICCs that Mplus produces are
L2/(L1+L2+L3) and L3/(L1+L2+L3). Note that there are two ICCs for each student level variable. If a variable is defined on the class level the ICC is computed the same way but L1=0.
 deana desa posted on Monday, April 08, 2013 - 6:02 am
Hi,

I would like to compare ICCs between two models which related to different specifications of between and within level weights.

Here is one part of the specification for model 1:

CLUSTER = idsc;
WEIGHT = tw;

and here's for model 2:
CLUSTER = idsc;
WEIGHT = L1w;
BWEIGHT = L2w;
WTSCALE = CLUSTER;
BWTSCALE = SAMPLE;


The ICCs derived from Mplus are as follows:

Model 1: Intraclass Correlation 0.417

Model 2: Intraclass Correlation 0.420


Can I know how can I verify these two values from mplus output? I tried to compute from the estimated values but I know that it's not right to do so.

I also saved the SIGB and SWM matrix files but how to read these file?
 Tihomir Asparouhov posted on Wednesday, April 10, 2013 - 2:28 pm
Deana

We dont have a way to compare these two estimates - however you can get a confidence interval for the ICC by specifying the unrestricted model in the model command and using a new parameter in model constraints to define the ICC as the new parameter.

Tihomir
 Izzelia Schwartz posted on Friday, April 12, 2013 - 10:24 am
Is it possible to get ICCs for clustered data using Type = Complex? The data used to create my latent variables are categorical. I have 61 clusters. Thank you.
 Linda K. Muthen posted on Friday, April 12, 2013 - 10:54 am
You need to use TYPE = TWOLEVEL BASIC to gett ICC's. You can do this for categorical variables. The ICC is for the underlying continuous latent response variable.
 deana desa posted on Tuesday, April 16, 2013 - 3:37 am
Thanks, Tihomir!

I tried the suggested approach like the following:

MODEL:
%within%
math1 (vw1);
%between%
math1 (vb1);
MODEL CONSTRAINT:
NEW(icc1);
icc1 = vb1 / (vb1 + vw1);

Now, the given ICC from the sample is .420 and the estimated ICC1 is as follows:

New/Additional Parameters
ICC1 0.428

So, these ICC and ICC1 are different intraclass correlations?

I think my question is how can I verify the ICC computed from the sample as outputed in Mplus (i.e. .420)?
 Linda K. Muthen posted on Tuesday, April 16, 2013 - 9:35 am
When you get the ICC from TYPE=BASIC, is math the only variable used?
 deana desa posted on Wednesday, April 17, 2013 - 2:46 pm
Linda,

Yes, math is the only variable for this null model.
 Bengt O. Muthen posted on Wednesday, April 17, 2013 - 3:03 pm
Can you please send the output to support for the run where you show the 2 values of 0.428 and 0.420.
 Robert Stewart posted on Tuesday, July 30, 2013 - 11:40 am
Hi. Is there a simple way to confirm, when computing intraclass correlations, if the default result is ICC(1,1) vs ICC(1,k)?
 Bengt O. Muthen posted on Tuesday, July 30, 2013 - 1:44 pm
What do you mean by those two expressions?
 rwstew posted on Tuesday, July 30, 2013 - 4:13 pm
By ICC(1,1) I mean a calculation based on a single rating to help determine how accurate a single rater may be, and by ICC(1,k) I mean a calculation based on the average of all raters within each group/cluster.

With the calculation described as BetweenVariance/(WithinVariance + BetweenVariance), I suspect the ICC given represents averages within each group.

Thank you for your help.
 Bengt O. Muthen posted on Tuesday, July 30, 2013 - 5:16 pm
Right.
 rebecca lazarides posted on Friday, January 24, 2014 - 9:03 am
Dear Drs. Muthén,
in my multi-level model I have an observed DV, 2 observed IVs and two latent IVs.

To calculate ICC for DV and IVs I computed with TWOLEVEL BASIC one model with regression equation (including latent factor part as well as regression part "DV on IV1 IV2") and one model without regression equation (including only the latent factor part "IV1 by x1 x2 x3" on both levels).

Average sample size as well as ICCs vary between both models.

Which one should be reported - the ICC from the model with regression equation or from the model without regression?

Thank you very much for your feedback.
 Linda K. Muthen posted on Friday, January 24, 2014 - 9:37 am
You should report the ICC's from TWOLEVEL BASIC. See the following FAQ on the website for further information:

Icc changes from one model to another
 rebecca lazarides posted on Saturday, January 25, 2014 - 9:12 am
Dear Prof. Muthén,

thank you very much for your quick response. Sorry for my misleading question - I used TWOLEVEL BASIC already as I read it in the statmodel threads about ICC.

What I would like to know is which ICC should be used (and reported) when testing whether multi-level analysis should be applied (in a first step before conduction multi-level SEM):

a) ICC from the model excluding all hypothezised regression paths
(e.g.
%within%
fw by x1 x2 x3;
y;
%between%
fb by x1 x2 x3;
w;

or

b) ICC from a model including all hypothezised paths
%within%
fw by x1 x2 x3;
fw on y;
%between%
fb by x1 x2 x3;
fb on w;

Thank you very much.
 Linda K. Muthen posted on Saturday, January 25, 2014 - 11:11 am
You should not use the ICC's from either of these models. You should use the ICC's from TYPE=BASIC based on the data not a model.
 rebecca lazarides posted on Sunday, January 26, 2014 - 3:27 am
Thank you very much.

Sorry for adding a follow-up question. I think your answer refers to ICC for observed variables. For this part of the model it helps me a lot.

However, I also would like to calculate the ICC for a latent factor and I think that I have to model this latent factor to get its' within variance and between variance for computation of its' ICC (variance within/total variance).

As I have to model the latent factor - should I model it with or without covariates?
 Linda K. Muthen posted on Sunday, January 26, 2014 - 10:37 am
I would use the model without the covariates. To do this, you should hold the factor loadings equal across between and within.
 rebecca lazarides posted on Monday, January 27, 2014 - 3:53 am
Great, thank you very much. This was very helpful.
 Sharon kwan posted on Friday, May 02, 2014 - 12:10 pm
Dear Prof. Muthén

May I know the standard ICC value for multilevel (two level) analysis in Mplus?
What is the justification of ICC in within and between levels? Can you please recommend some references to show the cut-off value for ICC that analyse using Mplus?

Thank you!
 Bengt O. Muthen posted on Friday, May 02, 2014 - 4:21 pm
There is no "standard" ICC value and no cut-off. Please study the paper on our website

Muthén, B. & Satorra, A. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267-316.
 Sharon kwan posted on Saturday, May 03, 2014 - 3:31 am
Thanks, prof.
 Jeroen de Jong posted on Tuesday, October 07, 2014 - 3:11 am
Dear Drs. Muthén,

I am testing a 1-2-model with a level 1 IV (employee work-family balance), a level 1 moderator (employee job autonomy), and a level 2 DV (academic reputation of the department). The average cluster size is 25 and the number of clusters is 44.

However, the ICCs of the IV and M are low (around .03). I understand the issues involving low ICCs in, for example, 2-1-models, but is it also problematic in 1-2-models? And if so, what could be a solution here? Aggregate the individual-level variables to Level 2?

Thank you!
Jeroen
 Bengt O. Muthen posted on Tuesday, October 07, 2014 - 9:17 am
An icc of 0.03 can be important when you have large cluster sizes. Try out the 2-level modeling.
Back to top
Add Your Message Here
Post:
Username: Posting Information:
This is a private posting area. Only registered users and moderators may post messages here.
Password:
Options: Enable HTML code in message
Automatically activate URLs in message
Action: