I'm trying to get clear on what the gold standard is in testing for intervention effects in longitudinal models. I would like some insights as to how to decide between simply treating intervention condition as a covariate (i.e., regressing the growth factors on the intervention dummy variable) or using a multiple group approach, in which a model that constrains the growth parameters to be equal between groups is compared to the model that allows them to vary, using a chi-square difference test. I am surprised how often these two approaches do not yield consistent results; in my experience, the multiple group model seems to detect intervention effects more often.
Also, what fit indices should be interpreted if using the multiple group approach? Thank you, Cindy Schaeffer
at the moment I'm also in this topic. Regarding multiple group modeling i would recommend you to read an article by "van Lier et. al, 2004". It seems to me, that multiple group modeling fits very well in the stepwise approach reccomended by muthen 2002 in his intervention related article. In other words, if you plan a mixture analyses of your intervention data, multiple group modeling is a nice and uncomplicated way to show overall effects, before doing mixture analyses with intevention as a covariate. if you expect no variances regarding the growth factors, i.e. you would like to do a "normal" latent growth model i would enter intervention status as a covariate in combination with some as covariates controlling for sex, SES and so on. But i would prefer to rely on Bengt and Linda, this was just my opinion ;-)
It is interesting to hear your experience, Cindy, that multiple group analysis more often finds intervention effects. This probably indicates that in some cases the covariate approach is too restrictive, for example in specifying equal growth factor variances for intervention and control groups. The multiple group approach was discussed in the Muthen-Curran 1997 Psych Methods paper. That was a special, more effective multiple group approach than a conventional one, where the intervention effect was singled out in the model. The approach focused on 3 intervention effects: main effect (on the mean slope), interaction effect, and effect on the variance of the slope. I am not sure it could be called a gold standard, but I would take this approach.
Thanks to you both - this was helpful. I'll take a look again at the articles you both mentioned. Looking at the citation dates, I'm either 3 or 10 years behind on this issue!!
Vanessa posted on Wednesday, June 30, 2010 - 7:26 pm
Hi, I'm planning to test for an intervention effect using the approach depicted in the Muthen-Curran 1997 Psych Methods paper. I'm new to LGCM (and Mplus) and have looked through your great notes on your site and various other papers, but still have a few queries regarding the order of the steps that need to be taken.
I have multiple indicators of my outcome construct, all assessed at 4 time points, in a control and treatment group. When testing for measurement invariance, is the appropriate approach to
1) test for invariance of loadings and intercepts across time, separately in the control and treatment group a) in measurement models (not LGCM)? b) and freeing up any eg. loadings that are not invariant (as indicated by modification indices)? then
2) test for invariance between groups, with all 4 timepoints specified a) but testing for invariance between groups, only for those parameters that showed invariance across time in the single group analyses? b) again in measurement models not LGCM?
3) run the two group LGCM but only constraining to equal those parameters that were invariant in the above
Apologies if the questions are basic, but there seems to be quite varying approaches to the preliminary steps in such models.
Vanessa posted on Thursday, July 01, 2010 - 10:57 pm
Thanks for the reply. Does it matter as to whether the tests for invariance are carried out in LGCM or in models that account for time simply by allowing all correlated residuals for same indicators at different time points and correlations between the factors?
The first step is to test for measurement invariance across time and across groups. After measurement invariance is established, you can use the latent variables in whatever type of model you want.
Vanessa posted on Tuesday, July 06, 2010 - 11:43 pm
Again thank you. Should the final 'invariance model' then form the basis for the LGCM? eg. should any residual covariances needed in the measurement model then be modeled in the LGCM?
I ask because covarying residuals were needed in my measurement model (additional to correlated factors for the measurement occasions), between all of the same indicators, at the different time points, in order to obtain good fit.
So, is it problematic if covarying residuals are allowed between all of the same indicators, at the different time points, in a LGCM?
I would start by testing what is most likely to hold. For example, if you think it is more likely that measurement invariance will hold across groups than time, I would start with groups.
Vanessa posted on Tuesday, July 13, 2010 - 5:06 pm
Thanks Linda. If I have strict (longitudinal) invariance in one group, and only strong in the other group, when I combine the groups' models to test for group invariance, should my final model include strict invariance across time for the group in which this holds (and not the other)?
Hello, I have 2 questions; 1) My question is about the loadings of the treatment growth factor for two-group LGM (Muthen 1997). Should it be the same with the linear growth factor loadings (eg.0 1 1.5 2.5) or as just fixing the first loading to 0 and others to 1 to introduce intervention after the first time point (as int BY bmi0@0bmi1@1bmi2@1bmi3@1)?
With the two-group LGM i found significant intervention effects on growth trajectories but when i use the intervention as a dummy coded covariate and regress the growth factors on it then all disappeared for all outcomes.Even the direction of the effect changed for some of them.
Is it because of the way of testing the intervention effect? Do you have any idea why this happens with the parallel process LGM? I appreciate a lot any idea/help on that problem.
1) That depends on your hypothesis for the treatment effect. Having the time scores 0, 1, 1, ... means that your treatment gives a jump right after it is introduced but the development has the slope (growth rate) as for the control group. Having the same say linear time scores as in the control group says that there is a gradual effect that alters the growth rate.
2) Seems like it should give the same results so it's hard to say without seeing your 2 runs - send input, output, data, and license number to firstname.lastname@example.org.
Vanessa posted on Thursday, January 12, 2012 - 10:28 pm
I'm following the two-group LGM approach for testing for intervention effects (Muthen & Curran 1997), using multiple indicators, with latent outcome factors for 4 timepoints.
I have a couple of queries:
1) some of the residuals for the outcome factors are negative (non-sig. though). Depending on which I constrain to be zero, the value (and significance) of the treatment slope mean changes.
a) Is it ok to constrain the negative ones to zero or is it best to constrain them all to be equal in each group (but not across groups)?
2) in some models, the treatment slope mean is significant in the non-standardised solution (eg. p=.002) and not significant in the standardised solution (eg. p=.15). This is a considerable difference - is this an issue and which would you report?
I would hold them all equal. This is what is typically done in multilevel modeling.
These differences occur because the raw and standardized coefficients have different sampling distributions. I would probably use the raw coefficient but I would also be conservative in that many tests are being performed so some type of Bonferroni correction should probably be made.
I have a question related to measurement invariance testing in pretest-post test data assuming change from intervention. My ultimate goal is to estimate intervention effect in latent means at post test. The latent trait is measured by 46 binary items. As a first step I compared groups on pretest and established measurement invariance across group. At post test I wasnít able to establish neither full nor partial measurement invariance across groups. Therefore I am wondering if measurement invariance across groups and time is an unrealistic goal in research involving subjects that are expected to change as a result of participation in the intervention. Treatment effects are evident at item level, for example items had very different difficulties across groups. To illustrate, for one item the difficulty in C group was 3.26 and in the treatment group was -1.44. Therefore I am also wondering if is reasonable to limit measurement invariance testing to pretest data only? Thank you, Anna-Mari
Perhaps the groups your refer to are treatment-control groups. Typically, with a factor model you want an intervention to affect the factor, not the items directly. It sounds like you have the latter situation, where direct effects from the treatment onto the post-test items makes for treatment-control group measurement non-invariance at that time point. That can also be modeled, however.
Thank you, Dr. Muthen, for your quick response. Yes, my groups refer to treatment and control group in a randomized control trial. Will you please elaborate on how the intervention affecting the items directly can be modeled or point me to references that can help me to understand this situation?
I need to add that there are significant differences between groups in factor means, if I hold item parameters equal across groups and across time. But as you pointed out I have measurement invariance at pretest only.
One approach is to impose post-test measurement invariance across treatment-control as a first step and then check Modification indices to see which items should relax the invariance. Those items experience specific treatment effects.
Jing Zhang posted on Tuesday, September 10, 2013 - 8:27 am
I am trying to fit a two-group growth model following the approach discussed in Muthen, 1997.
Would anyone share the relevant syntax for this model?
hi, we're analysing some data, whereby we used a school intervention (2 interventions and control) across 4 schools, over 3 timepoints. My understanding is that one approach to test whether the interventions are having an effect on the outcome, is to model this as an LGM.
Given all 3 conditions (2 interventions & 1 control) were collected at all 4 schools, I believe it would be best to consider whether there are any between effects at the school level.
So I was thinking of considering a model whereby the school level is the cluster, and checking to see whether the intraclass correlations are small, and if so then just considering the model without the schools as cluster.
Could you please let me know if this approach is correct, and whether this input (below) is appropriate to answer this question?
USEVARIABLES ARE t0 t1 t2 x school; USEV = x; CLUSTER = school; ANALYSIS: TYPE = TWOLEVEL; MODEL: %WITHIN% iw sw | t0@0t1@1t2@2; t0 t1 t2 (1); iw sw ON x; %BETWEEN% ib sb | t0@0t1@1t2@2; t0@0t1@0t2@0; ib sb ON x;
You can't really do 2-level analysis with only 4 clusters (schools). The recommended number is at least 20 and preferably many more than that.
You can create dummy variables for the 3 intervention categories, but it sounds like you don't have several schools per category so I don't see how intervention type and school can be disentangled. So the growth modeling is possible but assessing the intervention effect - assuming again that it is on the school level - may be hard.
So given the low number of schools, what would an appropriate approach be to establish homogeneity across the cluster level? My initial explorations suggest that the ICCs are very low, and therefore it may be feasible to only consider individuals over time (ignoring there are four schools). Note, that all three intervention conditions were present across each school.
In that instance, if you started with an unconditional model and established that the ICCs were low, then would it be possible to continue with the conditional analyses to test the effect of the intervention disregarding cluster?
Sounds like you have all 3 intervention conditions given in each of the 4 schools, so the intervention is perhaps best viewed as classroom-based.
So you can compare the 3 interventions within each of the 4 schools. And you could do a multiple-group analysis to see which parameters are the same across school. This means that school is viewed as a fixed, not random mode.
Thank you kindly for your responses. They have been most useful. I was hoping to ask, what would hopefully be the last questions relating to this topic as I learn more about it.
I have read the Muthen & Curran (1997) paper on using a two-group formulation, the added growth factor, for modeling intervention effects in LGMs and this seems more appropriate for what we are doing. In a two-group model for intervention studies (Figure 5), I understand that the added growth factor indicates the in/decrease beyond that of the control population.
My questions are in terms of the interpretation of the added growth factor, in particular the path from the treatment condition to the timepoints.
Is the correct interpretation that these are the mean differences at tx between control and intervention, after controlling for differences at t0 ? Put another way, does this model (Topic 4, pp 70-71) covary differences between the control and intervention at t0 for all subsequent comparisons?
The growth factor "t" is an added growth slope factor that influences the outcome means in the tx group after the first time point. So it represents an increasing or decreasing difference between the groups after the first time point.
thanks once again for your response. Given that the growth factor 't' is an added growth factor slopes, representing an in/decreasing difference between groups after the first time point, what approach would you advise to test mean differences at each timepoint between groups?
In addition to modelling growth change in the intervention group, as psychologists we are interested in whether the control and intervention group differ at each timepoint (t0, t1, t2). My understanding now, is that it is not as simple as interpreting the added growth factor at each timepoint, but was hoping there was a method to test this.
You would have to express the model-implied mean at each time point and for each group, as well as their differences, using Model Constraint. To express growth model means, see the handout for Topic 3 in our short courses online.
Thank you for your reply. In the following code, would Depress (my Y) represent the intercept (baseline level of Depression at Time = 0) for the control group (TxGroup = 0) while controlling for average Age and testing for the treatment effect?
DEFINE: Center Age (grandmean);
VARIABLE: Missing are all (-9999); Usevariable = ID TxGroup Age Time Depress; Cluster = ID; Within = Time; Between = TxGroup Age;
ANALYSIS: Type = Twolevel Random;
MODEL: %Within% S | Depress ON Time; %Between% S ON TxGroup Age; S ON Depress;
I have a few follow up questions about running a twolevel random model for an RCT, compared to SPSS mixed models.
1) Does Mplus provide estimates for the random intercept and random slope? When I run the code below, the output under Between includes Intercepts and Variances for S and Y, but I'm not sure what these represent. 2) In SPSS I can specify that a scaled identity covariance structure will be used for the random and repeated effects Ė how do you know or change which structure is being used in Mplus? 3) Should I include Y WITH S, and what would this indicate? Given there are covariates in the model does Y represent the average across time here? 4) The estimates for the covariates are the same in SPSS, but the estimate for Y ON TxGroup has the opposite sign (becomes negative). In Mplus, does this estimate represent the slope for the group coded as 0 (control) or 1 (treatment)?
MODEL: %Within% Slinear | GAD ON Time; %Between% Slinear ON TxGroup; !Treatment*Time interaction GAD ON Age Ethnicity TxGroup; !fixed effects
Thank your for your responses! To clarify I am understanding correctly, based on my output pasted below can I say that:
1) there is a significant random intercept for GAD and a significant random slope (output under Intercepts)
2) there was a significant effect of treatment (coded a 1) on the average level of GAD (my Y) but not on change over time/slope (Slinear)
3) covariance between repeated measurements is assumed to be 0, since there are no WITH statements in the output? Given my data is in long format (so I canít use y1 WITH y2), what code would specify a variance components/identity structure?
Slinear ON TxGroup b = .279 p = .681
GAD ON Age b =-.015 p = .881 Ethnicity b =-.446 p = .705 TxGroup b =-2.097 p = .039
Intercepts GAD b = 9.061 p = .002 Slinear b = -1.180 p= .017
1) The intercept for GAD is of interest only if the covariates have mean zero, that is, standardized for continuous covs and referring to the zero category for binary covs. Also, don't say "significant random intercept" - you should say "significant mean of the random intercept" because the random intercept is a variable, not a parameter.
2) Yes, if you add "controlling for age and ethnicity.
3) The covariance between repeated measurements is not zero because the random intercept and random slope makes them correlate. And identity type structure is obtained if you have a random intercept only.
Also, why don't you regress Slinear on Age and ethnicity as well?
I'm still a bit confused, because what I'm looking for is the equivalent variance and p-value for the random intercept and slope that the SPSS mixed models procedure provides in the output under estimates of covariance parameters 1) In Mplus, if GAD is significant under the Intercepts output, does this indicate there is variation between individuals in baseline values?
2) And if Slinear is significant under the Intercepts output, does this indicate that the slope varies between individuals?
You say: "the SPSS mixed models procedure provides in the output under estimates of covariance parameters"
I assume that you are talking about a run where you have covariates (conditional run as opposed to unconditional run). If so, what you are referring to is the variances and covariances among the residuals on level 2 (between) - so look for estimates labeled as such in the Mplus output.
If you send Support the SPSS output and the Mplus output for the same model, we can help you find the corresponding values.