I am testing for measurement invariance, and I had a question about testing for strict factorial invariance, where the item residual variances should be equal across groups (men and women for example). I have categorical items and am using WLSMV with the PARAMETERIZATION = DELTA option. If I do the following under this setting in the multiple groups (i.e. equalize the scalar factors across groups):
Does that have the same effect as equalizing the residual variances? Is this a correct way of testing strict factorial invariance (given that the constraints for configural, metric and strong factorial invariance are also incorporated in the model)? If no, what would be the correct way to test whether the residual item variances are equal between groups? How can I equalize the residual variances?
If you want to test for residual variance invariance, you should use the THETA parametrization where the residual variances are parameters in the model instead of the scale factors of the DELTA parametrization. The models that test this are a model with all residual variances fixed to one in all groups versus a model with residual variances fixed to one in one group and free in the other groups.
I want to do exactly the test you specified above: I want to compare a model which tests strong factorial invariance (intercepts and factor loadings free; residuals fixed to one in one group and free in the other 2 groups) with a model of strict factorial invariance with intercepts and factor loadings free and residuals fixed to 1 in each group.
Obs.Data is categorical; WLSMV estimator and Parameterization = THETA is used.
syntax differences: in the more restrictive model observed residuals (variable names "kd1";"kd2";etc.) are equal e.g., kd (1); kd (2); etc.
Result: The Chi square value is much lower for the strict model (and has less dfs) than for the less strict model in which residuals are free.
As i understand, a more restrictive model should generally have higher dfs and a higher Chi-square value, because there are less parameters to estimate which should yield in worse fit - or in the best case equal fit, indicating that the model is preferable (more parsimonious).
Questions: (1)May you help me with the problem described above?
(2)In Guide 6, you do not recommend testing strict factorial invariance with categorical data (p. 434). Why not?
(3)If i want to compare different models concerning invariance, i have to use DIFFTEST instead of comparing chi-values, right?
When you say intercepts and factor loadings are "fixed", I think you mean "held equal across groups". The stricter model with equal residual variances should not hold them equal and free across groups, but fixed at 1 in all groups.
A more restrictive model has more df's. It also has a worse, i.e. higher, fitting function value, where this value is found using Tech5 at the end of the iterations in the left-most column. With very ill-fitting models, chi-square is a less good gauge than the fitting function. DIFFTEST checks nesting using the fitting function values.
Testing of residual variance equality for categorical outcomes can be done, but since you often don't find equality and since it is not necessary for group comparisons of the factor means and variances, why do it?
Yes, DIFFTEST is the right approach.
Sascha posted on Monday, September 19, 2011 - 3:36 am
Thank you very much for your advice!
I am still wondering, if my results are correct, because the more restrictive model showed better fit indices (WLSMV and Theta Parameterization used in both models).
Question: I thought, that restricting a model should yield in an equal or worse fit ... Are CFI and RMSEA better because of parsimony or do you think there are mistakes in the syntax?
Data: Model 1: Configural invariance; factor loadings and intercepts free, latent means fixed in all groups; Result: Chi2=161.045, df=104;free par.=116;CFI=.975,RMSEA=.051, last Function-value in Tech5 output: 0.18299438D+00.
Model 2: Intercept Invariance; factor loadings and intercepts equal across groups; latent means fixed in one group. Chi2=170.101, df=115;CFI=.976,RMSEA=.047. Function-value in Tech5 output: 0.19829076D+00.
Model 3: Strict factorial invariance; factor loadings and intercepts equal across groups; latent means fixed in one group; equal residual variances (fixed to 1 in each group). Chi2=165.826, df=115;CFI=.978,RMSEA=.045. Function-value in Tech5 output: 0.25836231D+00.
Latent variances free in all models.
Difftest 1 vs. 2 is not significant: Chi2=22.294, df=23, p=.50 Difftest 1 vs. 3 is not significant: Chi2=53.159, df=43, p=.13
I have a run a CFA to test for strict factorial invariance of a 12 item list where all items have 3 ordinal categories. I have chosen the theta parametrization and used the WLSMV estimation method. Most fit indices point towards adequate fit of the strict factorial invariance model (Chi-Square/df = 1.6, RMSEA = 0.054, CFI = 0.981, TLI = 0.985). Moreover, the model fit is not worse when compared to the partial factorial invariance model where residual variances are not fixed (p = 0.0526 using DIFFTEST), albeit it borderline. My only concern is the large WRMR for the models: WRMR = 3.125 for the strict invariance model and WRMR = 2.813 for the partial invariance model. Ideally these should not exceed 1. Should I worry about these values? All modification indices are below 20 so not big improvements are to be expected by freeing some parameters.
I am trying to run a strict invariance with dichotomous variables and am having some difficulties. It SEEMS like I have everything right, but I'm getting the following error message: *** ERROR The following MODEL statements are ignored: and then it just lists the variables. I've played around and found I do not get the error message if I remove the rotation line from the syntax (cf-quartimax(oblique), I do not get the error message because Mplus gives no output whatsoever. I'm using the theta parameterization, wlsmv estimator but this odd error message; What is going on?
Eiko Fried posted on Wednesday, May 06, 2015 - 5:00 am
In addition to configural, metric, and scalar variance, we want to test residual/strict measurement invariance in a ESEM model. Using MLR, this simply means adding statements like this to the syntax: x1_time1 (a); x1_time2 (a);
However, we use the WLSMV estimator, and (using the theta parameterization) can't figure out a way this residual invariance model is nested in previous models. Any help would be appreciated. You stated above: "The models that test this are a model with all residual variances fixed to one in all groups versus a model with residual variances fixed to one in one group and free in the other groups."
But these are two models and not one strict invariance model? And neither of them seems nested in previous models (that do not specificy/estimate residual variances)?
Because you have to have one group with fixed 1 Theta residual variances, strict invariance is fixing all groups' residual variances at 1. Things are different with categorical outcomes than continuous outcomes; you lose some information.
Eiko Fried posted on Thursday, May 07, 2015 - 2:31 am
Bengt, what I don't understand is why constraining residuals to 1 in the strict invariance model leads to exactly the same model fit and DF as the strong invariance model.
Strong invariance: F1-F3 by H1_1-H17_1 (*t1 1); F4-F6 by H1_2-H17_2 (*t2 1); H1_1-H17_1 PWITH H1_2-H17_2; [F1@0]; [F2@0]; [F3@0]; [F4]; [F5]; [F6]; [H1_1$1] (1); [H1_2$1] (1); [... and so forth for all thresholds] chi^2 4299.74 with 532 DF
Strict invariance: Additionally: H1_1@1; H1_2@1; H2_1@1; H2_2@1; [... and so forth for all residuals] chi^2 4299.74 with 532 DF
If you look at the output for your strong invariance run I think you will see that the residual variances are all fixed at 1 (using WLSMV). You need to free them for the time point 2 items.
Eiko Fried posted on Thursday, May 07, 2015 - 9:25 am
Thanks Bengt. In the input for the strong invariance run in do not specify anything about the residual variances (apart from allowing them to be correlated over time with the pwith command). The output of the strong invariance model contains no information about residual variances at all (only item loadings & thresholds, factor means & variances).
If I rerun the strong invariance model, adding, H1_1@1; H1_2@1; (constraining the residual variance of one item to be the same across time)
I receive exactly the same chi-square and DF as in the model without these. But now the output contains the two residual variances (both at 1).
To me that means that Mplus constrains residuals by default to 1 across time. But that does not make much sense ... what am I missing?
WEAK INVARIANCE same except for constrained factor loadings F1-F3 by H1_1-H17_1 (*t1 1); F4-F6 by H1_2-H17_2 (*t2 1);
STRONG INVARIANCE same except latent factor means free at time 2 [F4]; [F5]; [F6]; additionally: - all thresholds constrained equal over time - all residual variances @1 time1, free time2 [!!!this leads to a model that is not nested!!!]
STRICT INVARIANCE same except for: - all residual variances @1
With categorical items, invariant factor loadings only (and not invariant thresholds) cannot be identified together with free residual variances (for time 2) unless your items have more than 2 categories. So in your binary case when you go from your weak invariance model, which doesn't have any free residual variances I assume, to your strong invariance model you add residual variances, causing the non-nested problem. We show how we do configural, metric, and scalar testing for categorical items in our 7.1 Language Addendum on our website, in the section Convenience Features. This relates to Millsap's work which we refer to.
So, the binary case does not fall neatly into the testing sequence of continuous items that you are attempting.
Eiko Fried posted on Friday, May 08, 2015 - 4:05 pm
Bengt, thank you so much for your insights.
We have the case of ordered-categorical items with 4 categories (and not dichotomous items). How does one build a nested strict invariance model in that specific case?
I read the language addendum information, but it doesn't feature a residual variance model which is required for measurement invariance (at least according to Meredith's highly influential paper).
Use the approach discussed in the language addendum for configural, metric, and scalar invariance with polytomous items. Then add to the scalar model the model where the residual variances are equal, which is the same as them being fixed at one in all instances.
Eiko Fried posted on Monday, May 11, 2015 - 10:02 am
Thanks Bengt. Following the procedure described, in the configural model, theta parameterization, I should——in addition to fixing all factor means to 0 and all residual variances to 1——set the factor variance of one or all factors (it's not clear for me from reading it) to 1. In both cases I receive the error for my ESEM models:
The variance of EFA factors in the reference block cannot be modified.
Because ESEM behaves like EFA, it already fixes the factor variances at 1.
Eiko Fried posted on Tuesday, May 12, 2015 - 4:42 am
Thanks. All 4 models converge, but all give 2 PSI warnings. With increasing constraints, the latent variable covariances more and more go above 1. In the strict invariance model, about 20% of all covariances of the 6 factors are above 1 (the covariance of one factor with itself is >4).
What is the standard way to deal with this problem?
Also, chi-square increases from configural to strong invariance, but then drops substantially in the strict invariance model. Can I ignore that because chi-square is not really interpretable with WLSMV, or is that reason to be concerned? (DFs all increase, and all DIFFTESTs are significant)
No. It is more a matter of trying to modify the measurement instrument, e.g. adding items. Not all measurement instruments are well-fitted by a factor model, no matter how many factors you choose. I assume you are using EFA/ESEM so that the reason for high factor correlations isn't the usual CFA reason.
Eiko Fried posted on Wednesday, May 13, 2015 - 2:51 am
Yes we're using EFA/ESEM. We're analyzing 4 different rating scales for depression (16-30 items), and since 1-7 factor solutions have been reported for each in the last 40 yrs, choosing an a priori CFA structure is difficult (even when doing it data-driven, over 50% of all items have high cross loadings).
The scales are not really "psychometric" measures; clinicians made up items, and for some reason the instruments are still used as gold standard half a century later.
So I guess it is not surprising that no proper factor structures emerge. Thanks for the help!