I'm trying to get clear on what the gold standard is in testing for intervention effects in longitudinal models. I would like some insights as to how to decide between simply treating intervention condition as a covariate (i.e., regressing the growth factors on the intervention dummy variable) or using a multiple group approach, in which a model that constrains the growth parameters to be equal between groups is compared to the model that allows them to vary, using a chi-square difference test. I am surprised how often these two approaches do not yield consistent results; in my experience, the multiple group model seems to detect intervention effects more often.
Also, what fit indices should be interpreted if using the multiple group approach? Thank you, Cindy Schaeffer
at the moment I'm also in this topic. Regarding multiple group modeling i would recommend you to read an article by "van Lier et. al, 2004". It seems to me, that multiple group modeling fits very well in the stepwise approach reccomended by muthen 2002 in his intervention related article. In other words, if you plan a mixture analyses of your intervention data, multiple group modeling is a nice and uncomplicated way to show overall effects, before doing mixture analyses with intevention as a covariate. if you expect no variances regarding the growth factors, i.e. you would like to do a "normal" latent growth model i would enter intervention status as a covariate in combination with some as covariates controlling for sex, SES and so on. But i would prefer to rely on Bengt and Linda, this was just my opinion ;-)
It is interesting to hear your experience, Cindy, that multiple group analysis more often finds intervention effects. This probably indicates that in some cases the covariate approach is too restrictive, for example in specifying equal growth factor variances for intervention and control groups. The multiple group approach was discussed in the Muthen-Curran 1997 Psych Methods paper. That was a special, more effective multiple group approach than a conventional one, where the intervention effect was singled out in the model. The approach focused on 3 intervention effects: main effect (on the mean slope), interaction effect, and effect on the variance of the slope. I am not sure it could be called a gold standard, but I would take this approach.
Thanks to you both - this was helpful. I'll take a look again at the articles you both mentioned. Looking at the citation dates, I'm either 3 or 10 years behind on this issue!!
Vanessa posted on Wednesday, June 30, 2010 - 7:26 pm
Hi, I'm planning to test for an intervention effect using the approach depicted in the Muthen-Curran 1997 Psych Methods paper. I'm new to LGCM (and Mplus) and have looked through your great notes on your site and various other papers, but still have a few queries regarding the order of the steps that need to be taken.
I have multiple indicators of my outcome construct, all assessed at 4 time points, in a control and treatment group. When testing for measurement invariance, is the appropriate approach to
1) test for invariance of loadings and intercepts across time, separately in the control and treatment group a) in measurement models (not LGCM)? b) and freeing up any eg. loadings that are not invariant (as indicated by modification indices)? then
2) test for invariance between groups, with all 4 timepoints specified a) but testing for invariance between groups, only for those parameters that showed invariance across time in the single group analyses? b) again in measurement models not LGCM?
3) run the two group LGCM but only constraining to equal those parameters that were invariant in the above
Apologies if the questions are basic, but there seems to be quite varying approaches to the preliminary steps in such models.
Vanessa posted on Thursday, July 01, 2010 - 10:57 pm
Thanks for the reply. Does it matter as to whether the tests for invariance are carried out in LGCM or in models that account for time simply by allowing all correlated residuals for same indicators at different time points and correlations between the factors?
The first step is to test for measurement invariance across time and across groups. After measurement invariance is established, you can use the latent variables in whatever type of model you want.
Vanessa posted on Tuesday, July 06, 2010 - 11:43 pm
Again thank you. Should the final 'invariance model' then form the basis for the LGCM? eg. should any residual covariances needed in the measurement model then be modeled in the LGCM?
I ask because covarying residuals were needed in my measurement model (additional to correlated factors for the measurement occasions), between all of the same indicators, at the different time points, in order to obtain good fit.
So, is it problematic if covarying residuals are allowed between all of the same indicators, at the different time points, in a LGCM?
I would start by testing what is most likely to hold. For example, if you think it is more likely that measurement invariance will hold across groups than time, I would start with groups.
Vanessa posted on Tuesday, July 13, 2010 - 5:06 pm
Thanks Linda. If I have strict (longitudinal) invariance in one group, and only strong in the other group, when I combine the groups' models to test for group invariance, should my final model include strict invariance across time for the group in which this holds (and not the other)?
Hello, I have 2 questions; 1) My question is about the loadings of the treatment growth factor for two-group LGM (Muthen 1997). Should it be the same with the linear growth factor loadings (eg.0 1 1.5 2.5) or as just fixing the first loading to 0 and others to 1 to introduce intervention after the first time point (as int BY bmi0@0bmi1@1bmi2@1bmi3@1)?
With the two-group LGM i found significant intervention effects on growth trajectories but when i use the intervention as a dummy coded covariate and regress the growth factors on it then all disappeared for all outcomes.Even the direction of the effect changed for some of them.
Is it because of the way of testing the intervention effect? Do you have any idea why this happens with the parallel process LGM? I appreciate a lot any idea/help on that problem.
1) That depends on your hypothesis for the treatment effect. Having the time scores 0, 1, 1, ... means that your treatment gives a jump right after it is introduced but the development has the slope (growth rate) as for the control group. Having the same say linear time scores as in the control group says that there is a gradual effect that alters the growth rate.
2) Seems like it should give the same results so it's hard to say without seeing your 2 runs - send input, output, data, and license number to email@example.com.
Vanessa posted on Thursday, January 12, 2012 - 10:28 pm
I'm following the two-group LGM approach for testing for intervention effects (Muthen & Curran 1997), using multiple indicators, with latent outcome factors for 4 timepoints.
I have a couple of queries:
1) some of the residuals for the outcome factors are negative (non-sig. though). Depending on which I constrain to be zero, the value (and significance) of the treatment slope mean changes.
a) Is it ok to constrain the negative ones to zero or is it best to constrain them all to be equal in each group (but not across groups)?
2) in some models, the treatment slope mean is significant in the non-standardised solution (eg. p=.002) and not significant in the standardised solution (eg. p=.15). This is a considerable difference - is this an issue and which would you report?
I would hold them all equal. This is what is typically done in multilevel modeling.
These differences occur because the raw and standardized coefficients have different sampling distributions. I would probably use the raw coefficient but I would also be conservative in that many tests are being performed so some type of Bonferroni correction should probably be made.
I have a question related to measurement invariance testing in pretest-post test data assuming change from intervention. My ultimate goal is to estimate intervention effect in latent means at post test. The latent trait is measured by 46 binary items. As a first step I compared groups on pretest and established measurement invariance across group. At post test I wasn’t able to establish neither full nor partial measurement invariance across groups. Therefore I am wondering if measurement invariance across groups and time is an unrealistic goal in research involving subjects that are expected to change as a result of participation in the intervention. Treatment effects are evident at item level, for example items had very different difficulties across groups. To illustrate, for one item the difficulty in C group was 3.26 and in the treatment group was -1.44. Therefore I am also wondering if is reasonable to limit measurement invariance testing to pretest data only? Thank you, Anna-Mari
Perhaps the groups your refer to are treatment-control groups. Typically, with a factor model you want an intervention to affect the factor, not the items directly. It sounds like you have the latter situation, where direct effects from the treatment onto the post-test items makes for treatment-control group measurement non-invariance at that time point. That can also be modeled, however.
Thank you, Dr. Muthen, for your quick response. Yes, my groups refer to treatment and control group in a randomized control trial. Will you please elaborate on how the intervention affecting the items directly can be modeled or point me to references that can help me to understand this situation?
I need to add that there are significant differences between groups in factor means, if I hold item parameters equal across groups and across time. But as you pointed out I have measurement invariance at pretest only.
One approach is to impose post-test measurement invariance across treatment-control as a first step and then check Modification indices to see which items should relax the invariance. Those items experience specific treatment effects.
Jing Zhang posted on Tuesday, September 10, 2013 - 8:27 am
I am trying to fit a two-group growth model following the approach discussed in Muthen, 1997.
Would anyone share the relevant syntax for this model?
hi, we're analysing some data, whereby we used a school intervention (2 interventions and control) across 4 schools, over 3 timepoints. My understanding is that one approach to test whether the interventions are having an effect on the outcome, is to model this as an LGM.
Given all 3 conditions (2 interventions & 1 control) were collected at all 4 schools, I believe it would be best to consider whether there are any between effects at the school level.
So I was thinking of considering a model whereby the school level is the cluster, and checking to see whether the intraclass correlations are small, and if so then just considering the model without the schools as cluster.
Could you please let me know if this approach is correct, and whether this input (below) is appropriate to answer this question?
USEVARIABLES ARE t0 t1 t2 x school; USEV = x; CLUSTER = school; ANALYSIS: TYPE = TWOLEVEL; MODEL: %WITHIN% iw sw | t0@0t1@1t2@2; t0 t1 t2 (1); iw sw ON x; %BETWEEN% ib sb | t0@0t1@1t2@2; t0@0t1@0t2@0; ib sb ON x;
You can't really do 2-level analysis with only 4 clusters (schools). The recommended number is at least 20 and preferably many more than that.
You can create dummy variables for the 3 intervention categories, but it sounds like you don't have several schools per category so I don't see how intervention type and school can be disentangled. So the growth modeling is possible but assessing the intervention effect - assuming again that it is on the school level - may be hard.
So given the low number of schools, what would an appropriate approach be to establish homogeneity across the cluster level? My initial explorations suggest that the ICCs are very low, and therefore it may be feasible to only consider individuals over time (ignoring there are four schools). Note, that all three intervention conditions were present across each school.
In that instance, if you started with an unconditional model and established that the ICCs were low, then would it be possible to continue with the conditional analyses to test the effect of the intervention disregarding cluster?
Sounds like you have all 3 intervention conditions given in each of the 4 schools, so the intervention is perhaps best viewed as classroom-based.
So you can compare the 3 interventions within each of the 4 schools. And you could do a multiple-group analysis to see which parameters are the same across school. This means that school is viewed as a fixed, not random mode.