Hello everyone, I recently saw a presentation on what was, essentially, multilevel measurement invariance. The authors suggested that strict equality constraines could not be imposed across levels, as in regular invariance tests. Instead, the authors suggested a type of proportional invariance, where item 1's loading could be any given proportion of item 2's loading, but this proportion should hold across levels. OR, they suggested one could propose that the between-groups loading for item 1 be an observed proportion of the within-groups loading of item 1, but that this proportion hold for all items.
Clearly, this is a more complex way to set equality constraints than are usually allowed in Mplus. However, would constraints such as these be possible? Could one, for example, state that a factor loading or a structural relationship be proportionally equivalent across levels? If so, I think this could be a new way to get the word out about Mplus: It allows tests of multilevel measurement and structural invariance!
bmuthen posted on Saturday, August 13, 2005 - 2:47 pm
I haven't seen this kind of argument, but maybe it arises when letting all loadings be free and held equal across levels - setting the metric of the factor by having the factor variance fixed at one. Because the within and between variances are different this would call for proportionality constraints rather than equality constraints. I would instead set the metric by a factor loading fixed at one and then holding the remaining loadings equal across levels, letting the within and between factor variances be free. But perhaps I am overlooking something.
To answer your question about proportionality constraints, I think this can be done in Mplus using Model Constraint. For example, in abbreviated form,
fw by y2 (p2w) y3 (p3w);
fb by y2 (p2b) y3 (p3b);
p2b = c*p2w; p3b = c*p3w;
Here, c is a dummy parameter which is the proportion. c can be estimated by giving it a label connected with a dummy parameter. Such dummy parameter handling can be done in the current version, but will be easier in Version 4. I can tell you more about it if my response is on the right track.
Hi Bengt, This is exactly the kind of modeling constraint I am imagining. Please do tell me more about how to estimate c by giving it a label connected with a dummy parameter.
Also, regarding your initial sentence and example constraints, do you think this kind of model would only work when allowing loadings to be free but set to proportional invariance across levels? Could it not also be used by allowing the items to interrelate in a given way at one level and having the same interrelation at another level?
%Within% fw by y2 (p2w) y3 (p3w);
%Between% fb by y2 (p2b) y3 (p3b);
p2w = c*p3w; p2b = c*p3b;
Here, c would specify a proportion but do so with a different meaning than in your example. Do you see any problems with doing one vs. the other? They seem to address different questions.
Mplus is great, I can't wait for version 4!
bmuthen posted on Monday, August 15, 2005 - 9:35 am
Proportionality for any kind of parameters would be possible in this approach.
The dummy parameter specification in the current Mplus is done by labeling a parameter that would not in itself be identified, for example,
such that this loading has no impact on the model or its likelihood. Note that fdum should also have zero correlation with other factors. Then, "c" in the Model Constraint paragraph has a place in the parameter array - and is identified since the Model Constraint statements place constraints on the parameters that c has to try to match.
Again, sorry to bother, but one real question is: If a parameter "c" is freely estimated, do you think that the reduction in the degrees of freedom for the model (-1) because of estimating the parameter is justified?
In other words, because the parameter doesn't influence the model, for difference testing could df be upwardly adjusted by +1 to pretend that no parameter was estimated?
bmuthen posted on Sunday, August 21, 2005 - 9:15 am
The model with C appearing in Model Constraint expresses the 2 B1, B2 slopes in terms of just 1 parameter, C (and other already existing parameters). Therefore this model has 1 fewer (unrestricted) parameters and should have 1 more degree of freedom. Difference testing of this C constraint should have 1 df.
MarcReis posted on Tuesday, January 03, 2006 - 7:48 am
Hello Dr. Muthén,
regarding the test of measurement invariance across levels as you described above (setting the metric by fixing the loading of item 1 to 1.0 and constraining the others to be equal across levels): Is this kind of test still appropriate, if (a) it´s necessary to fix some non-significant between-residuals to zero and/or (b) allowing correlated residuals on the within- but not on the between-level?
In this case, the models wouldn´t be exactly the same on both levels. I would appreciate a comment on this, thanks!
It is usually the case that a different number of factors are found at the within and between levels of a multilevel model. And even when there are, the factors most likely have a different meaning for the two levels. The suggestion that I see from the previous postings is that holding the factor loadings equal across levels is most likely not appropriate.
MarcReis posted on Wednesday, January 04, 2006 - 2:19 am
Many thanks for the answer, but I'am unsure whether my question was not clear enough. It`s not that I want to impose an equality constraint for the factor loadings across levels a priori. As a result of a MFA, I found that a model with the same factor structure between and within fits the data very well (however, I had to allow some correlated residuals within and fix some non-significant residuals between to zero). Additionally, judging by the MLR-chi²-difference test as described on the homepage, there is no significant worsening in the model fit, when I impose the equality constraint on the factor loadings across levels. As far as I understand, this is mentioned in the Metha & Neale - article (2005). So, does your previous answer fits to this situation?
I think correlated residuals on within probably represent a hidden factor. I suspect that you have not actually done an EFA on the pooled-within and Sigma between matrices as a first step to see how many factors there are on each level. If you have not, I would start there. I think it would be hard to support that the factors are the same when you have to add residual covariances on the within level.
MarcReis posted on Wednesday, January 04, 2006 - 12:15 pm
Actually I did initial EFA and the accordingly specified CFA/MFA fits well in terms of CFI, RMSEA, etc., but there are some high MI. I will see whether the residuals are really necessary, thanks a lot for your time!
I’d like to investigate measurement invariance in a multilevel context using multiple group models. I have factors on the within and between level; some of them have continuous and some have categorical factor indicators.
For continuous indicators, I adopted a suggestion by Thompson and Green (2006) to test for both invariance of factor loadings and intercepts. Invariance of factor loadings is tested first by analyzing the covariance structure only, not the mean structure. Having explored invariance of loadings, invariance of intercepts is tested next.
For step 1 (covariance structure only) I decided to extract the estimated sigma within and sigma between matrices for the groups to be compared and to analyze within and between separately.
1a) Is it correct that intercepts of the outcomes are only of interest on the between level and analyzing a mean structure to test invariance of intercepts on the within level does not make sense? 1b) If yes, is it sufficient to check for invariance of the loadings on the within level?
2) Since means of factor indicators are given on the between level, would it be correct to first analyze the between covariance matrix only (using the estimated sigma between), then "add" the means (given in the output for a type=twolevel analysis) to the covariance matrix used as input and then test for intercept invariance?
Invariance for categorical outcomes causes even more trouble to me. As far as I know, threshold invariance has to be analyzed along with invariance of loadings.
Within level: 3) Is it possible to use an extracted correlation matrix to check for measurement invariance - maybe in the same way this is done for continuous outcomes?
Between level: As far as I know, a between matrix cannot be extracted for categorical outcomes. So I thought about using type=twolevel. 4a) Is this starting setup correct: - defining the factor on the between level while having no variables on the within level if the factor indicators are measured on the between level - defining the factor on the between level while allowing just for indicator correlations on the within level if the indicators are measured on the within level
Next step would be to do a comparison of models a) with thresholds and loadings fixed between groups and factor means freely estimated with the exception of the mean in one group fixed to zero and b) with thresholds and loadings freely estimated and the means in all groups fixed to zero. 4b) Is this specification correct? 4c) If yes, how would I compare these models? I'm not sure if they are nested. Besides a direct comparison: What differences in values for thresholds and (standardized) loadings would be troubling?
The answer to all of your questions is yes with the following additional information. For categorical outcomes, within and between matrices are not available at this time. You can create a data set where each observation is a cluster and analyze that. The models you mention are nested.
1) If each observation is a cluster, this is (in case of a binary outcome) a multilevel logistic regression where probability of endorsement is a function of loading and threshold on lvl1 and of factor score on lvl2? 2a) If yes, invariance for within structure could be tested by a cross-lvl interaction of group membership (dummy vars on lvl2) on the lvl1 regression coefficients with lvl 2 being persons? 2b) For invariance of between structure, it would be analogous with the units of between structure (in my case school classes) on lvl2?
I think Linda's answer referred to your 4a) statement: - defining the factor on the between level while having no variables on the within level if the factor indicators are measured on the between level.
That is, you will then have a single-level factor model for your between variables, where the clusters is your sample.
Nikolai Eton posted on Thursday, September 04, 2008 - 5:35 am
I am relatively new to multilevel modeling and would appreciate it a lot if you can give me a reference on testing measuring and factor invariance across levels. Furthmore, how can this be applied in Mplus? By using the "model constraints" command?
I believe I am getting the same error message as user Michael J. Zyphur above. I used proportionality constraints (p2d, p3d, and p4d; and p2d, p3d, and p4d) which load onto two different latent slopes.
I employed the "Model constraint" section of code to define the proportionality constraints.
But I receive the error message: *** ERROR in MODEL command Unknown variable(s) in a BY statement: (P2D)
Do I need to define the constraint variables in the USEVARIABLES statement Portions of my code are below.
Thank you, Lisa
!level and slope for depression d0 by ldep1@1; d1 by ddep2 (p2d) ddep3 (p3d) ddep (p4d); !<---- proportionality constraints d0; d1; [d0 d1]; d0 with d1*.25;
!level and slope for bmi b0 by lbmi1@1; b1 by dbmi2 (p2b) dbmi3 (p3b) dbmi4 (p4b);! <---- proportionality constraints b0; b1; [b0 b1]; b0 with b1*.25;
Juliette posted on Wednesday, March 28, 2012 - 6:00 am
Hello, I am trying to understand if it is possible/acceptable to test measurement equivalence across two time points in a multilevel CFA model. I have come across some responses regarding measurement equivalence across groups in a multilevel model, but I am not clear how this applies to equivalence across time. If it is possible to do, do you go through the same steps as in single level measurement invariance testing (allowing all to vary but fixing the factor variances to one and means to zero respectively, constraining factor loadings, constraining loadings and intercepts across time)but do this within each level? Thank you.
See the Topic 4 course handout on the website under multiple indicator growth. Testing for measurement invariance across time is shown here. This can be done only if the data are in wide multivariate format.
Juliette posted on Friday, March 30, 2012 - 3:55 am
Thank you for pointing me in the right direction. So, I follow the steps outlined in the handout, testing for invariance in the within part and the between part of the model simultaneously? Or do you have to do this for the within part and the between part separately so that you are testing for invariance at one level at a time? Thank you.
I just have one more question. When I tried it togheter (like I described before) the Xi2 difference is significative, supporting the hypothesis. However, when I tried each path independently the differences are not significative.
Can I constrian all the paths togheter and compare it with the unconstrained model to support the hypothesis?
The overall test likely has more power than the individual tests. The way you approach this should take into account your hypotheses. If you have specific hypotheses, you should test those. Otherwise, you can use an overall test.
4. Another question, please. When carrying out config, metric, scalar, and structural invariance tests, in each case, I constrain the factor means to equal "0" under the MODEL command. Lastly, I use the savedata command to save the output of the structural test (DIFFTEST=structural.dat). Next, to test for the equivalence of latent factor means, as I understand it, I am supposed to constrain the latent means of one group to zero. To do this, do I now omit the "[f1@0]" command from under the MODEL command and place this restriction under the MODEL GROUP2 command?
Dear Drs Muthen, We are trying to run a multi-level (2 levels, pupils within schools)model with a pupil variable measured at 2 time points using the same measure with 10 categorical (0,1,2) items. To achieve measurement invariance over time we have constrained factor loadings and thresholds to be equal over time as illustrated below !time 1 EMO1 BY X1_T1 (1) X2_T1 (2) !time 2 EMO2 BY X1_T2 (1) X2_T2 (2)
Excluding the multi-level component the model runs. When we include the multi-level bit the following error message appears.
*** ERROR in MODEL command The thresholds of variables on the between and within levels exist only on the between level.
We are not sure how to proceed. The multi-level model runs if the threshold constraint is removed, so can we run it without constraining thresholds over time? I should also note here that we do not have any level-2 predictors and are using a multi-level approach to account for school-level variation.
Thank you! The model runs when I follow your suggestion and move just the thresholds to the between level. Just for my own understanding can you please explain to me why the threshold constraints are between and the rest of the measurement model within? (or direct me to somewhere that does).
I will be fitting a multilevel SEM to examine the influence of how a change in school-level variables from elementary school to middle school affects children's academic and behavioral outcomes in middle school. But first I'd like to just examine the measurement models for the school-level variables (one for elementary schools and one for middle schools). Unfortunately, there are only about 20 elementary schools and 90 middle schools, which makes things tough in terms of having large enough Ns in the measurement models, particularly for elementary schools. However, since the school-level variables will eventually have children "below" them in the SEM, can I artificially increase my Ns when I test the school-level measurement models by duplicating the set of school-level variables as many times as there are children for that school and then fitting a CFA model to those augmented data sets? So instead of having one set of observations for each school on the school-level variables, I have N_j, where N_j is the number of children that were in school j when those school-level variables were collected? If not, is there a better way to handle this small-N measurement-model problem? Thanks very much!
I'm afraid that wouldn't be correct if the school-level variables are truly measurements of schools, not students. With only 20 observations you can't investigate a measurement instrument, so you have to rely on past research to form a scale.
I have a sample of individuals nested in teams across three conditions. I ran a multi-group MCFA to test for measurement/structural invariance across the conditions. When I change the referent group/condition in the models, I get almost identical results (fit indices, etc.) across both the configural model and the metric model. However, I get very different fit indices and parameter estimates across referent groups in the scalar model. Any idea why this might be happening?
As part of a scale validation project, I recently performed CFA and IRT analyses of a multilevel data set of 63 3-category ordinal survey items originating from 1181 participants from 22 villages. For the CFAs I used TYPE=COMPLEX due to the modest number of villages and found good support for a 7-dimension correlated factors CFA. However, a reviewer of our manuscript feels we haven't done enough to validate the construct at the village level. ICCs for the subscales were modest, ranging from .02 to .15.
The reviewer believes the low ICCs in conjunction with the analysis approach taken indicate insufficient validation of the scale at the group level. Given the moderate number of villages, my co-authors and I are curious if there are additional validation techniques available in Mplus that we could consider or whether we're stuck concluding that while there is solid support for the 7-dimension construct at the person level, support at the community level is limited? For instance, could the newer alignment methods be tried to help address these issues? Any suggestions appreciated.