 Computing effect sizes of of growth r...    Message/Author  David Myers posted on Thursday, April 06, 2000 - 11:46 am
Imagine we have a growth model that is a quadratic, then the rate of learning can be expressed as S + 2Qt where S is the linear term, Q is the quadtratic term, t is time (this is just the partial derivate of outcome--test scores in this case, with respect to time). Now, I want to be able to compute the variance of learning. It would seem that I could take the predicted factor scores for S and Q, for each child i, form the quanity S+2Qt for each child, and then compute the variance of the new variable. Is this correct?

I want to be able to use this to compute effect sizes where the numererator of the effect size is the difference in learning rates for two groups of children. Thanks in advance.  bmuthen@ucla.edu posted on Friday, April 07, 2000 - 9:29 am
That would be one way of getting the variance - through estimated factor scores. This variance, however, may have the usual biases due to estimating factor scores. How about the alternative of computing

V(S + 2Qt) = V(S) + 4tt V(Q) + 4t Cov(S,Q)

where those variances and covariance are the growth model estimates?  Julia Kim posted on Tuesday, April 11, 2000 - 2:11 pm
We've come across an instance where the equation above yields a negative variance. Specifically, a growth model for Math achievement produced the following:

V(S) = 322.341
V(Q) = 10.746
COV(S,Q) = -59.359

When t = 3, V(S + 2Qt) = -3.

What may be the implications for our model? Thanks.  b1uesman44 posted on Wednesday, November 23, 2005 - 10:10 am
This is an old post, but you might notice that for these parameter estimates the correlation between S and Q is out of range. I've computed

CORR(S,Q)=-1.008568461

Such results are usually an indication of model misspecification, or a potential overspecification. It seems that the Q term is not needed (multicollinearity).

Estimated covariance matrices are not required to follow the rules associated with correlation matrices, and sometimes negative estimates of variance, out of range correlations, etc., can occur.

Perhaps you can use constraints, and/or boundaries so that your covariance parameters result in correlations that fall between -1 and +1.

Then this formula should be positive.  bmuthen posted on Wednesday, November 23, 2005 - 6:39 pm
Good point. And note that centering at the average time might make the s, q correlation go down.  Bill Dudley posted on Friday, October 03, 2008 - 2:02 pm
I have data from a study in which slopes differ for treatment and control - as estimated in a Latent growth curve analysis. The variables are similar to those in my next study. I would like to use the estimates from the successful model in a Monte Carlo and model different effects sizes and N�s (vary the slopes and NOBS)

In �How to use Monte Carlo �� Muthen & Muthen 2002, the effect size is defined as the group difference in slopes divided by the standard deviation of the slope growth factor.(page 7). In the growth model monte carlo set up on page 17 we see that the intercept and slope are set at 0 and .2 [i*0 s*.2]. Because S is regressed onto X we are provided with the residual variance of s is .09. s*.09;RIGHT??

Given my confusion on this, I am at a loss about how to use my current analyses to model power in my next study - ----

Q1 I assume that the standard deviation of the slope is computed from the parameters in this model OR perhaps derived from the preceding unconditional l model. Page 16. If the SD is computed can you point me to the formula for this so that I might see how to vary to effect sizes

Q2 Your Monte carlo examples use variables with means of zero. Can I just use the mean and SD�s from my real data or is there an advantage to outcomes with mean = 0 ? or should I convert my raw data to scores for this purpose?  Bengt O. Muthen posted on Saturday, October 04, 2008 - 12:04 pm
Q1. You compute the SD(s) from sqrt{Var(s)},

Var(s) = b^2*Var(x) + resvar(s);

where b is the slope in the regression of the slope on the covariate x (b^2) is the square of b). You get this in Tech4. You can also define effect size as the outcome mean difference (ctrl-tx)/SD at a certain time point - might be more down to earth.

Q2. Keep your variables as they are. The means of the growth factors are given as in the Topic 3 handout, but also in Tech4.  Bill Dudley posted on Wednesday, October 29, 2008 - 11:29 am
Please excuse the hiatus between postings, however I'd like to follow up my power estimation post of 10/03. Consider a test of differences in slopes between two groups as modeled in the growth2 example from Muthen & Muthen 2002. I think that this is comparable to the test of the time by group interaction in a Repeated measures ANOVA. If I use GPOWER to estimate the effect size for this interaction with 50 per group, alpha = .05 and correlations between measures a .3, I get an f ~.15. If I use Growth2 from Muthen and Muthen for a group difference in linear slopes I can detect an effect size of .63. It seems likely that the GPOWER estimate is far too liberal. I assume that this discrepancy is multiply determined � in the ANOVA we are dealing with Cohen�s f and in the LCM I assume that we are working Cohen�s d (??). Also I assume that in the Growth2 model that Y1 � y4 are not correlated whereas they are in the GPOWER estimation. And I assume that the Growth2 model is far more restrictive than the more general interaction test in GPOWER in that the Growth2 model is for a linear change in the positive direction etc. Can you comment on this?  Bengt O. Muthen posted on Wednesday, October 29, 2008 - 6:52 pm
I would think GPOWER (which I am not familiar with) considers effect size in terms of the means of the outcomes for a certain time point, not the random slope means. In the Mplus Monte Carlo setting you can also consider effect size for the outcome at a certain time point - expressing the y mean difference divided by the SD. The growth model does have correlated y's because the y's are all influenced by the growth factors. Tech4 says how strongly correlated they are. You are probably right that the linear growth model gets a different power estimate than the repeated measures ANOVA which doesn't impose linear growth.  socrates posted on Tuesday, June 30, 2009 - 7:22 am
I developed a GMM of treatment courses in which I regressed the variance of the slopes on type of treatment within each of the latent classes. Now, I am confused how to derive effect sizes from the slopes. According to Raudenbush & Xiao-Feng (2001), effect sizes can computed by the formula:

mean slope of treatment 1 minus mean slope of treatment 2 divided by the population standard deviation of the polynomial trend of interest (tau).

So, is tau the square root of the residual variance of the slope in the Mplus output?

Thanks  Linda K. Muthen posted on Wednesday, July 01, 2009 - 8:39 am
Yes.  Karl Hallmackenreuther posted on Thursday, February 18, 2010 - 8:22 am
I'm analyzing a randomized trial over 4 years (4 measurement points, the first is pre-test) via LGM and want to compute the effect size as "d" [estimated outcome mean difference (ctrl-tx)/SD at the last time point]. I've heard that one should use the pre-test SD to calculate d for the last time point in LGMs. That seems not very comprehensible to me and in my special case SDs for the outcome are heavily increasing over time for both control and intervention. So I prefer to use SDs pertaining to the last measurment point to get a smaller but more "realistic" d. What is your opinion?  Bengt O. Muthen posted on Thursday, February 18, 2010 - 9:50 am
I agree with you. If one is interested in the mean difference at the end of a study, it seems natural to consider the variation at the end of the study. One argument against that is that this variation is affected by treatment, so you should work with baseline variance. But even the control group can change variance from baseline to the end. My question would more be - if at the end the control and treatment group variances are very different, which should we use? In that case, perhaps the control group variance at the end is more meaningful because that refers to the normative development without treatment.  Karl Hallmackenreuther posted on Thursday, February 18, 2010 - 10:54 am
I was using the estimated outcome variance at the last measurement point because I was calculating "d" for estimated means and thought this would be more stringent. This estimated variance was equal across intervention/control because I modeled Intervention as a dummy-covariate. There were rather small variance residuals and observed variances seemed comparable across intervention and control. Is it o.k. to use the estimated variation in this case?  Bengt O. Muthen posted on Thursday, February 18, 2010 - 11:14 am
Yes, that's what I would recommend.  Cindy Schaeffer posted on Tuesday, February 23, 2010 - 7:43 am
I too need to compute effect sizes for a tx effect (S regressed on dichotomous tx variable 0 = control, 1 = intervention) in LGMs with 5 unequal time points, continuous outcomes, and 2 additional covariates (age and gender). I think I am clear on how to do this, but have a few questions:

1. Do any of the procedures outlined in these posts change when the Tscores option is used (TYPE = random missing)? I ask because I am getting huge "d" values, 3+ in some cases when S on tx p values are in the .05 range. This is due in part because I have very small residual variances on S in the denominator.

2. What about when outcomes are count variables? (Tscores option and TYPE = random). I am getting excessively large effect sizes on these in particular.

3. Is there a way to look at overall effect size across 5 time points rather than difference in mean at the final time point? Isolating the change effect to the last time point seems to undermine the advantage of using repeated measures in the first place.

Thanks much,
Cindy  Bengt O. Muthen posted on Tuesday, February 23, 2010 - 2:10 pm
1. I assume "d" is the effect size? I think it is more understandable to look at effect size as a y mean difference (see also 3.), rather than an s difference. The variance of s can be small even when the y variance is large, which makes the s difference less important.

2. For counts you don't have a y variance parameter, so I am not sure what the effect size definition for y would be.

3. I agree that this should be done. It is really the area between the tx and ctrl curves that is important. Not sure off hand how to best quantify that, however.  Hanno Petras posted on Friday, March 12, 2010 - 11:15 am
Hi,

I am stuck understanding the computation of effect sizes from the Muthen & Muthen (2002) paper (page 7). In your paper, you say that a slope effect of 0.2 translates into an effect size of 0.63 and that the effect size is computed as the ratio of the slope difference divided by the standard error. Here is my computation and I appreciate your advice, because I am stuck.

Slope| x=0 = 0.2
Slope| x=1 = 0.2 + 0.2*0.460 where 0.460 is the mean of x
To compute the variance of the slope I used:

SQRT (Var(s))=b^2 *var(x) + resvar(s)
==> SQRT((0.04 * 0.248) + 0.09)=0.316

0.292-0.2/0.316=0.291.

Thanks,

Hanno  Linda K. Muthen posted on Saturday, March 13, 2010 - 9:48 am
slope mean|x=0 = a
slope mean|x=1 = a + .2

The difference is .2

.2 / Sqrt (.04*.25 + .09) = .63  Hanno Petras posted on Saturday, March 13, 2010 - 3:17 pm
Hi Linda,

Hanno  Leah Vermont posted on Saturday, October 15, 2011 - 1:21 pm
Hello.

I am running a parallel process LGM.
The correlation coefficients between the respective slopes (both linear and quadratic) of my two constructs were positive and significant. I would like to know how best to determine the magnitude of these significant correlations, in order to determine the effect sizes. Specifically, which parameter estimates should be used?

Thank you for you time,
LNV  Bengt O. Muthen posted on Saturday, October 15, 2011 - 6:18 pm
Just look at the TECH4 output, where you find the correlations (in contrast, the results section gives the covariances). I don't know that you can use the term "effect size" in the context of a correlation.  Karen S. Mitchell posted on Friday, August 24, 2012 - 10:05 am
Hello,

I've estimated a two-level linear growth model (not latent) to test the impact of an intervention on scores across 9 time points. I have one treatment group and one control group and need to calculate an effect size. Would I use the formula from Feingold (2009)?

Beta_11 / (square root of tau)

Where Beta_11 is the unstandardized coefficient of the regression of the slope on treatment (0/1). Would I take the square root of the slope residual variance for the denominator?  Bengt O. Muthen posted on Friday, August 24, 2012 - 10:29 am
It isn't clear what one should use for effect size in growth models. You refer to the effect size on the slope in which case the denominator SD should be based on the slope variances, not the slope residual variance (so see TECH4). But one could also argue for the effect size on the outcome itself at a certain time point, so you need to express the outcome model-estimated mean at that time point and the model-estimated variance at that time point. Our Topic 3 handouts show how to compute those.  Linda K. Muthen posted on Friday, August 24, 2012 - 10:34 am
See slides 97 and 98 of the Topic 3 course handout on the website.  Karen S. Mitchell posted on Friday, August 24, 2012 - 11:06 am
Thank you. I'm using type=random (see below), so Tech4 isn't available. Is there another way to obtain the slope variance in this model?

Usevariables are
dgroup Y person time ;

Cluster = person ;
Within time ;
Between = dgroup ;

Missing are .;

ANALYSIS: TYPE = TWOLEVEL RANDOM;

MODEL:
%WITHIN%
s | Y on time ;

%BETWEEN%
Y s on dgroup ;
Y with s ;  Bengt O. Muthen posted on Friday, August 24, 2012 - 11:11 am
You have on Between

s ON dgroup;

which implies that the s variance is easily computed as

V(s) = b^2 * V(dgroup) + res var (s),

where b is the model-estimate slope in s ON dgroup that you see in the output, V(group) is obtained as the sample variance from Sampstat, and res var (s) is the model-estimated residual variance of s in the output.  Karen S. Mitchell posted on Friday, August 24, 2012 - 11:37 am
Thanks!  antonio zuffiano posted on Friday, March 29, 2013 - 7:17 am
Dear Bengt and Linda

First of all, thank you so much for your wonderful support.

I am still confused about the computation of effect size in LGM and I have to answer to one reviewer who asked me information about effect size.

I used an LGM (three time points: pre-test, 6-month post-test, 12-month follow-up) in which I regressed on the latent non-linear slope (coded 0 * 1) of my outcome Y the following variables:

"treatment" (0 = control group, 1 = intervention group);
"sex" (0 = male, 1 = female);
"age of participants";
"the latent intercept" which represents the initial status.

I found a positive and significant effect of "treatment" on the slope and I would like to summarize the size of this effect.

Can you help me?

Best

Antonio  Bengt O. Muthen posted on Friday, March 29, 2013 - 4:04 pm
I think the most down to earth effect size is to compute the effect size in the outcome at the last time point as implied by the growth model. That is, using the difference in model-implied estimated mean for control and intervention groups divided by the SD for the outcome at the last time point. The SD is obtained directly in the output (use RESIDUAL) and the means are computed from the estimates in the output in line with slides 97-98 of our Topic 3 handout on our web site.  Ping Kuo posted on Saturday, December 19, 2015 - 4:33 am
Hello,

My model is a longitudinal model with only two assessments. How could I calculate the effect sizes for latent mean difference? For example, latent mean=0 at Time 1 and latent mean at Time 2 was freely estimated. I got latent mean at Time 2= -.450 and factor variance at Time 1 and Time 2 were.77, 1.1, respectively. Does the value of factor mean from "STDYX Standardization" indicate effect size?
Thanks a lot.  Bengt O. Muthen posted on Sunday, December 20, 2015 - 5:27 pm
Divide -0.450 by 0.77 if you want the answer in the time 1 SD.  Hye Jeong Choi posted on Wednesday, April 13, 2016 - 6:50 am
Hello,

I have two questions. I have four different groups (e.g., four different linear trajectories) Some groups have significantly different slope or intercept.

Q1. How can I calculate effect size between group1's slope and groups2's slope?
I assumed that mean difference from two groups' slope divided by growth factor SD. Am I correct? If so, which groups' growth factor's SD should I use? They have different slope and different growth factor' SD.

Q2. How can I calculate effect size intercept difference among groups? I searched articles but I couldn't find it. Do you have a formula for this?  Hye Jeong Choi posted on Wednesday, April 13, 2016 - 6:54 am
Hello,

I have two questions. I have four different groups (e.g., four different linear trajectories) Some groups have significantly different slope or intercept.

Q1. How can I calculate effect size between group1's slope and groups2's slope?
I assumed that mean difference from two groups' slope divided by growth factor SD. Am I correct? If so, which groups' growth factor's SD should I use? They have different slope and different growth factor' SD.

Q2. How can I calculate effect size intercept difference among groups? I searched articles but I couldn't find it. Do you have a formula for this?  Bengt O. Muthen posted on Wednesday, April 13, 2016 - 12:53 pm
Please post these questions on SEMNET.  Susan Collins posted on Thursday, May 25, 2017 - 12:28 pm
And a follow-up question: If the residual variance for slope in a linear growth model is negative (which I am now holding constant), how can one generate the effect size for the slope? I understand the formula is dependent on the resvar(s).  Susan Collins posted on Thursday, May 25, 2017 - 12:33 pm
Oops--looks like I forgot to post my original question. I apologize. I am conducting a monte carlo study for a power analysis. I understand how to calculate cohen's d for the differences in slope for a linear growth model. How could one do this in a growth model where the measured outcomes are counts but (I assume) the latent variable is continuous? Thanks!  Bengt O. Muthen posted on Thursday, May 25, 2017 - 6:46 pm
By effect size for the slope I assume that you consider slope mean values for say 2 different groups and divide by the slope SD. That same approach would be used for count outcomes. You want to modify a model with negative variance estimates.  Aurelie Lange posted on Wednesday, January 09, 2019 - 5:04 am
Dear Dr Muthen,

I have a linear growht model with three time points. We are simply analysing change over time.
Is it possible to compute an effect size for the slope coefficient? I.e., is it possible to provide an estimate of the size / relevance of the change over time?

Thank you!  Bengt O. Muthen posted on Wednesday, January 09, 2019 - 5:02 pm
See the paper

Feingold, A. (2018). Time-varying effect sizes for quadratic growth models in multilevel and latent growth modeling. Structural Equation Modeling: A Multidisciplinary Journal. DOI: 10.1080/10705511.2018.1547110  Yaqiong Wang posted on Wednesday, June 19, 2019 - 4:12 pm
I'm running parallel process latent growth curve modeling with constructs A and B, both of which were measured for four times. Construct B follows a quadratic developmental trajectory while contract A has a linear trajectory. The regression coefficients of slopes B (both linear and quadratic)on intercept A are significant.

1. Could you tell me if/how I can calculate the effect size for the regression coefficient?

2. If I run another model with constructs C and B, can I use the effect size of each model to compare the strength of the effect between constructs A and C?

Thanks!  Bengt O. Muthen posted on Saturday, June 22, 2019 - 6:26 am
1. How about using the STD solution where both the IV i and the DV s are standardized.

2. Sounds ok.    Topics | Tree View | Search | Help/Instructions | Program Credits Administration