ICC in Mplus PreviousNext
Mplus Discussion > Multilevel Data/Complex Sample >
Message/Author
 Student 09 posted on Sunday, February 15, 2009 - 3:22 pm
Hi

sometimes it is of interest to inform readers/reviewers what has been termed a "residual ICC" - the ratio of between-group variance/total variance in a dependent variable which remains after covariates have been entered into the model.

However, I just noted that the (presumed) residual ICC of my dependent variable (I use Mlwin example data) does not differ from the according ICC I requested via 'multilevel basic', even though I added several significant group-level covariates. At the same time, the between-group R_square of the model with covariates clearly increased as compared to the null model without any between-group covariates.

Is it correct that the ICC for a dependent variable shown by Mplus ALWAYS relates to a null model without covariates?

Thanks for your answer!
 Bengt O. Muthen posted on Monday, February 16, 2009 - 6:33 am
Yes, you should instead compute your residual ICC from the model-estimated parameters.
 Nidhi Kohli posted on Wednesday, October 05, 2011 - 12:38 pm
Hello,

I have a two-level nested cross-sectional data where the dependent variable is nominal with 3 unordered categories. The dataset contains within- and between-level covariates. I want to compute ICC (between variance/total variance) for the twolevel model with and without covariates. I tried running the model with TYPE=TWOLEVEL BASIC, but the output did not contain the ICCs. Can you please help me to compute ICC in Mplus.

Thanks!!
 Bengt O. Muthen posted on Wednesday, October 05, 2011 - 8:57 pm
There is no within-level variance with a nominal outcome (this parameter does not exist) and therefore icc's are not defined. I would just see if the between-level variances for the random intercepts of the nominal variable are significant.
 Nidhi Kohli posted on Thursday, October 06, 2011 - 8:59 am
Thank you for your response. I have a follow-up question. I did not see the between-level variances for the random intercepts of the nominal variable in the output. How can I get the between-level variances for the random intercepts of the nominal variable?

Thank you very much.
 Nidhi Kohli posted on Thursday, October 06, 2011 - 9:18 am
I am sorry, I forgot to mention that I was not able to run TYPE=TWOLEVEL BASIC when I classified the dependent variable as nominal. Thus, I classified the dependent variable as categorical so that at least I can get something in the output, which obviously didn't work. What is the best way to get the between-level variances for the random intercepts of the nominal variable? I greatly appreciate your help.
 Linda K. Muthen posted on Thursday, October 06, 2011 - 9:39 am
To get the between-level variance of the random intercept of the nominal variable, you need to mention the variance in the between part of the model. If the nominal variable is u with three categories, say

u#1;
u#2;
 Nidhi Kohli posted on Thursday, October 06, 2011 - 10:22 am
Just to confirm, are you referring to model with TYPE=TWOLEVEL?

Another question that I have is: will the random intercepts of u#1 and u#2 be treated as continuous variables? If yes, then the regression coefficients at the second -level obtained from regressing u#1 and u#2 on the between-level covariates are the linear regression coefficients? Thanks.
 Linda K. Muthen posted on Thursday, October 06, 2011 - 1:22 pm
Yes.

Yes and yes.
 Nidhi Kohli posted on Monday, October 10, 2011 - 8:57 am
Hello Linda,

One last question on this thread. I fitted a Twolevel model where I requested the program to estimate the between-level variance of the random intercept of the nominal variable by using the following command:

%BETWEEN%

alt_dep#1 alt_dep#2 ON md_age md_female md_white img family panel_pts;

alt_dep#1;
alt_dep#2;

The output shows that the residual variances corresponding to alt_dep#1 and alt_dep#2 is non-significant (p-value>=0.05). Based on this information, is it reasonable to assume that a Twolevel model is not required. Alternatively, there is not much variability at the second-level, hence, a single-level model is appropriate for this dataset?

Thanks.
 Li Lin posted on Monday, October 10, 2011 - 12:29 pm
Hello,
I have a question about two-level model with dichotomous respones. For each observation and each endogenous variable, I'd like the residual (observed-predicted) for further analysis. Are there any output options I can specify to save these residuals?

Thanks for your help.
 Linda K. Muthen posted on Monday, October 10, 2011 - 3:23 pm
Nidhi: It would seem so.
 Linda K. Muthen posted on Monday, October 10, 2011 - 3:25 pm
Li Lin: No options are available for this. You would need to create the predicted score and the difference score in DEFINE.
 Li Lin posted on Thursday, October 13, 2011 - 6:55 am
Hi Linda
Thanks for your response. I have two more questions.
1. How to create the predicted score in Mplus?
2. I have dichotomous responses in the model. residuals = observed-predicted score? Does multivariate normality assumption hold for the residuals?
 Linda K. Muthen posted on Thursday, October 13, 2011 - 11:52 am
1. Is the dependent variable binary? Are you using WLSMV or MLR?

2. See TECH10 and the technical appendix Standardized and Normalized Residuals.
 Li Lin posted on Thursday, October 13, 2011 - 12:37 pm
Yes the dependent variables are binary. I'm using MLR.

Thanks!
 Linda K. Muthen posted on Thursday, October 13, 2011 - 1:26 pm
You need to use DEFINE to compute a log odds for each person:

logodds = -a + bx

where a is the threshold and b is the regression coefficient.

The probability is equal to

1 / (1 + exp (-logit)

where - logit is a - bx.

If a person has a probability greater than .5, assign a value of 1 as the estimated value. If a person has a probabiity less than or equal to .5, assign a value of 0 as the estimated value. Then take the difference between the observed and estimated values.
 Sarah Phillips posted on Thursday, July 12, 2012 - 10:56 am
Hello,

I am sorry for this basic question. I have been reading the discussion board and looking at the Topic 5 handout you suggest, but I am confused about how to calculate the overall ICC for my model so that I can calculate the design effect.

Here's what I am doing. Could you help me figure out what to do differently?

1) I estimate what in HLM would be the null model-- just my dependent variable-- using type=basic twolevel and specifying the dependent variable only on the within portion of the model-- i.e.

Analysis:
Type=basic twolevel;

Model:
%Within%
engage by m_a3 m_a31 m_a60 m_a72 m_a74 m_a79 m_a80;
m_a3 with m_a31 m_a60;
m_a31 with m_a60;

%Between%

In the output, I get ICCs for Y variables. But I am not sure how to combine them to get the total ICC. Do I average the ICCs for the Y variables to get the ICC for the null model? Do I use some other option to get the total residual variance for the within and between? Something else?

Thanks so much for your continued help.

Best,

Sarah
 Bengt O. Muthen posted on Thursday, July 12, 2012 - 6:31 pm
I am not familiar with the concept of "overall" or "total" icc. To me, icc is a variable-specific concept.

You can also computed an icc for a latent variable as we describe in Topic 7.

You say that you want to compute a design effect. I wonder why. Design effects are sometimes used to make an approximate correction of SEs of estimates. But there is no need to do that since you can simply use Type=Complex to get the right SEs.
 Sarah Phillips posted on Friday, July 13, 2012 - 6:46 am
Thanks, Dr. Muthen. Understanding that in MPLUS icc is variable-specific helps.

I'm computing the design effect to determine if I need to use multilevel modeling or can ignore the clustering.

Based on your response, I should be computing the design effect for each variable individually?

-Sarah
 Linda K. Muthen posted on Friday, July 13, 2012 - 5:04 pm
It is not necessary to compute design effects. You should run the analysis without taking clustering into account and with TYPE=COMPLEX that takes clustering into account. Compare the standard errors. If they are very close, it is not necessary to take clustering into account.
 Sarah Phillips posted on Sunday, July 15, 2012 - 11:57 am
Thanks Dr. Muthen!
 Thula_Chelvan posted on Monday, August 20, 2018 - 1:54 pm
Hello,

I'm working with multilevel data (within = time for variable y, between1 = subject, between2= therapist), and I'm hoping to calculate the ICC to examine the variance related to therapists (i.e., therapist effect). The data is in long form.

Would you be able to confirm that the below code is appropriate to estimate the ICC? Thanks!

USEVARIABLES = y therapist subject;
MISSING = ALL (999);
CLUSTER = therapist subject;

ANALYSIS:
ESTIMATOR = ML;
PROCESSORS = 4;
TYPE = THREELEVEL RANDOM;

MODEL:

%WITHIN%
y;

%BETWEEN subject%
y;

%BETWEEN therapist%

y;
 Bengt O. Muthen posted on Monday, August 20, 2018 - 2:27 pm
Yes, but see also the FAQ on our website "ICCs for 3-level" using Threelevel Basic:

http://www.statmodel.com/download/ICCs%20for%203-level.pdf
 Thula_Chelvan posted on Monday, August 20, 2018 - 3:37 pm
Hi Dr. Muthen,

Thank you. I read the link you shared and I am a little confused.

Is threelevel basic another way to find the ICCs via the formulas B2/Total and B3/Total? Is the syntax I shared earlier still appropriate?

Thanks in advance
 Bengt O. Muthen posted on Monday, August 20, 2018 - 3:46 pm
Yes on both.
 Jilian Halladay posted on Sunday, February 09, 2020 - 8:21 am
Hi There,

How do you calculate the ICC for a three level probit model? I have put the variance of the outcome at the upper levels so I obtain the variance at level 2 and 3 but there is no variance output for level 1. I know if logit models, level 1 is fixed at 3.29. Is there a similar fixed variance for probit models? If so, what it is?

Thanks so much
Jillian
 Bengt O. Muthen posted on Sunday, February 09, 2020 - 11:04 am
It is 1 for probit.
 Loes Abrahams posted on Thursday, February 20, 2020 - 11:34 pm
Dear Prof. Muthen,

I noticed that the ICCs that Mplus provides when estimating a two-level intercept only model are slightly different than the ICCs that I calculated by hand using the between-person variance/(between-person variance+within-person variance) that is shown in the output. Could you give me some more information on how this is possible? Also, is it in this case recommended to report the (estimated) ICCs that Mplus provides, or is it better to calculate them myself based upon the between- and within-person variances?

Thank you in advance!
 Bengt O. Muthen posted on Friday, February 21, 2020 - 3:53 pm
See the FAQ on our website:

ICC changes from one model to another
 Yvonne Terry-McElrath posted on Thursday, June 25, 2020 - 7:55 am
Dear Drs. Muthen -

I am running a 2-level model (TWOLEVEL RANDOM, MLR estimator, montecarlo integration, using BWEIGHT) with a 3-level nominal outcome. As recommended above to N. Kohli (2011), I want to see if the between-level variances for the random intercepts of the nominal variable are significant in a model with no other parameters:

MODEL:
%WITHIN%;
%BETWEEN%
alcuse#1;
alcuse#2;

When I use the outcome in the original form (1,2,3) then the variance estimates are 2.354 (SE 0.725, p=0.001) for #1, and 0.483 (SE 0.212, p=0.023) for #2.

When I reverse code the outcome (i.e., it is again coded 1,2,3, but the cases identified as “1” and “3” have been reversed) the variance estimates are 2.424 (SE 0.799, p=0.002) for #1, and 1.931 (SE 0.426, p=0.001) for #2.

Question 1: I had understood threshold #1 to indicate essentially the intercept between the “1” and “2” of the outcome, and threshold #2 to indicate essentially the intercept between the “2” and “3” of the outcome. But if this was the case, then the larger variance estimate would belong to #2 in the reverse-coded model?

Question 2: Since the variable was simply reverse coded, why are the variance estimates not equal in magnitude?

Many thanks - Yvonne
 Bengt O. Muthen posted on Thursday, June 25, 2020 - 10:30 am
This is caused by the change of the reference class. In the first approach the reference class is 3 (the last class) but in the second approach it is 1 (in terms of the original scoring). The analogous thing happens to the means, so it is not variance specific.
 Yvonne Terry-McElrath posted on Thursday, June 25, 2020 - 11:05 am
Dr. Muthen –
Thank you so much for your response, and clarifying that I was misunderstanding the threshold reference category.

I have one further question. I know that if I was using a dichotomous outcome, I would be able to calculate the ICC from the obtained between-level variance estimate by using MODEL CONSTRAINT (based on Raykov & Marcoulides 2015 doi: 10.1177/0013164414564052).

Is it defensible to do the same thing with the obtained variances of my 3-category nominal outcome? For example:

MODEL:
%WITHIN%
;
%BETWEEN%
modhidbn#1 (p1);
modhidbn#2 (p2);
MODEL CONSTRAINTS:
new ICC1 ICC2;
ICC1 = p1/(p1+(3.14159^2/3));
ICC2 = p2/(p2+(3.14159^2/3));

Thank you so much.
Yvonne
 Bengt O. Muthen posted on Thursday, June 25, 2020 - 4:23 pm
I am not aware of ICCs for nominal outcomes. You could rerun the model as binary with one category versus all others - so one run for each category. Then the formulas would be right. You may also want to take a look at our Short Course Topic 7 where we talk about 2-level logistic regression. We describe ways to quantify how important the between-level variation is and that may generalize to the nominal case. See also how this is described in the 2-level LCA paper by Henry & Muthen on our website which does consider a nominal DV (a latent class variable).
 Yvonne Terry-McElrath posted on Friday, June 26, 2020 - 5:26 am
Thanks so much.
Back to top
Add Your Message Here
Post:
Username: Posting Information:
This is a private posting area. Only registered users and moderators may post messages here.
Password:
Options: Enable HTML code in message
Automatically activate URLs in message
Action: