 Monte Carlo for Multilevel Analysis    Message/Author  Yaacov Petscher posted on Thursday, November 01, 2012 - 12:48 pm
Greetings - I am in the process of simulating power for a randomized control trial while account for the measurement error of the manifest variable. For a single level trial, I have been able to corroborate power estimates using both Mplus and Optimal Design. However, when moving to a multilevel illustration with a single manifest outcome, I'm running into difficulties. Assuming an ICC of 10% at the between level, the results suggest power of .45; however, Optimal Design suggests power of .36. I'm wondering if this, indeed, a true estimate or whether my code is not accounting for something else.

montecarlo:
names are y w;
cutpoints=w(0);
nobservations = 100;
ncsizes=1;
csizes = 10 (10);
seed = 58459;
nreps = 1000;
between = w;

ANALYSIS:
TYPE IS TWOLEVEL;
processor=8;
estimator=ml;

MODEL POPULATION:
%Within%
fw by y@1;
y@.01;
fw*1;

%Between%
w@1;
fb by y@1;
y@0;
fb on w*.5;
fb*.1;

MODEL:
%Within%
fw by y@1;
y@.01;
fw*1;

%Between%

fb by y@1;
y@0;
fb on w*.5;
fb*.1;  Bengt O. Muthen posted on Thursday, November 01, 2012 - 9:18 pm
2 quick thoughts:

- do you really have icc=0.10? Isn't the between variance 0.35 and the within variance 1, which means icc=0.35/(1+0.35) = 0.26.

- with only 10 clusters the SEs' will be poorly estimated and therefore the Monte Carlo estimated power will be poorly estimated.  Yaacov Petscher posted on Friday, November 02, 2012 - 6:12 am
Thank you for the response! When I ran the code here the ICC for Y came up as .096. So is it a reasonable hypothesis that the power for the Monte Carlo, though poorly estimated due to relatively few clusters, may more accurately reflect the power than Optimal Design which does not account for the residual variance?  Bengt O. Muthen posted on Sunday, November 04, 2012 - 11:12 am
If the population, icc=0.26 and you get icc=0.096 for your sample, then the estimate isn't very good - and that is probably due to few clusters. I would not trust the Monte Carlo power estimates in such a case.  Yaacov Petscher posted on Tuesday, November 06, 2012 - 6:58 am
Thank you, Bengt. One final issue I'm curious about is how you came up with the between-variance of .35.  Bengt O. Muthen posted on Tuesday, November 06, 2012 - 11:14 am
The fb on w regression seems to give between variance:

0.5*0.5*1 + 0.1 = 0.35

Within variance

1+ 0.01 = 1.01.

icc = 0.35/(0.35+1.01) = 0.26  Yaacov Petscher posted on Tuesday, November 06, 2012 - 12:53 pm
Apologies for the novice question, but is there a reference you may be able to point me toward for those underlying equations? I had assumed that the total variance was the 1 + .01, so that the between variance was .01/(.01+1) and the within was 1/(.01+1).

In larger cluster size conditions (20, 30, 50, 100) I used this convention for the ICCs and effect sizes, and I was able to obtain converging estimates of power between Mplus and Optimal Design.  Bengt O. Muthen posted on Tuesday, November 06, 2012 - 1:12 pm
Total variance = between + within.

The Raudenbush-Bryk multilevel book should have these formulas.  Yaacov Petscher posted on Wednesday, November 28, 2012 - 8:52 am
Hi Bengt - One final follow-up as I've been working through this. In the code from this thread the between variance, as I understand, is .1 from the fb*.1 code, and the within variance is 1 from the fw*.1 code. From this I thought the ICC would be .10 (between) / .1 + 1 (between + within).

In your previous response saying that the beta weight is factoring into the estimation of the between variance? Is the general form then B^2*between variance+within variance?  Bengt O. Muthen posted on Wednesday, November 28, 2012 - 11:55 am

fb on w*.5;
fb*.1;

do not imply that the fb variance is 0.1. It is the residual variance of fb that is 0.1. The variance of fb also consists of the variance due to the w influence, just like in regular regression.    Topics | Tree View | Search | Help/Instructions | Program Credits Administration