Strict Factorial Invariance in multig... PreviousNext
Mplus Discussion > Confirmatory Factor Analysis >
Message/Author
 Abdel posted on Tuesday, April 26, 2011 - 7:26 am
Dear all,

I am testing for measurement invariance, and I had a question about testing for strict factorial invariance, where the item residual variances should be equal across groups (men and women for example).
I have categorical items and am using WLSMV with the PARAMETERIZATION = DELTA option.
If I do the following under this setting in the multiple groups (i.e. equalize the scalar factors across groups):

{item1*1} (1);
{item2*1} (2);
{item3*} (3);
{item4*1} (4);
{item5*1} (5);

Does that have the same effect as equalizing the residual variances? Is this a correct way of testing strict factorial invariance (given that the constraints for configural, metric and strong factorial invariance are also incorporated in the model)?
If no, what would be the correct way to test whether the residual item variances are equal between groups? How can I equalize the residual variances?
 Linda K. Muthen posted on Tuesday, April 26, 2011 - 7:32 am
If you want to test for residual variance invariance, you should use the THETA parametrization where the residual variances are parameters in the model instead of the scale factors of the DELTA parametrization. The models that test this are a model with all residual variances fixed to one in all groups versus a model with residual variances fixed to one in one group and free in the other groups.
 Abdel posted on Tuesday, April 26, 2011 - 7:47 am
Thank you very much! How do I then specify the residual variances? (I know that scale factors are specified by {item1*1} ).
 Linda K. Muthen posted on Tuesday, April 26, 2011 - 9:45 am
Residual variances are specified as:

item1;
 Abdel posted on Wednesday, April 27, 2011 - 2:53 am
Thank you so much!
 Sascha Wüstenberg posted on Thursday, September 15, 2011 - 2:06 am
I want to do exactly the test you specified above:
I want to compare a model which tests strong factorial invariance (intercepts and factor loadings free; residuals fixed to one in one group and free in the other 2 groups) with a model of strict factorial invariance with intercepts and factor loadings free and residuals fixed to 1 in each group.

Obs.Data is categorical; WLSMV estimator and Parameterization = THETA is used.

syntax differences:
in the more restrictive model observed residuals (variable names "kd1";"kd2";etc.) are equal
e.g., kd (1); kd (2); etc.

Result: The Chi square value is much lower for the strict model (and has less dfs) than for the less strict model in which residuals are free.

As i understand, a more restrictive model should generally have higher dfs and a higher Chi-square value, because there are less parameters to estimate which should yield in worse fit - or in the best case equal fit, indicating that the model is preferable (more parsimonious).

Questions:
(1)May you help me with the problem described above?

(2)In Guide 6, you do not recommend testing strict factorial invariance with categorical data (p. 434). Why not?

(3)If i want to compare different models concerning invariance, i have to use DIFFTEST instead of comparing chi-values, right?
 Sascha Wüstenberg posted on Thursday, September 15, 2011 - 2:08 am
In the first part it should mean:
intercepts and factor loadings fixed - in both cases (strict and strong invariance).
 Bengt O. Muthen posted on Thursday, September 15, 2011 - 9:38 am
When you say intercepts and factor loadings are "fixed", I think you mean "held equal across groups". The stricter model with equal residual variances should not hold them equal and free across groups, but fixed at 1 in all groups.

A more restrictive model has more df's. It also has a worse, i.e. higher, fitting function value, where this value is found using Tech5 at the end of the iterations in the left-most column. With very ill-fitting models, chi-square is a less good gauge than the fitting function. DIFFTEST checks nesting using the fitting function values.

Testing of residual variance equality for categorical outcomes can be done, but since you often don't find equality and since it is not necessary for group comparisons of the factor means and variances, why do it?

Yes, DIFFTEST is the right approach.
 Sascha posted on Monday, September 19, 2011 - 3:36 am
Thank you very much for your advice!

I am still wondering, if my results are correct, because the more restrictive model showed better fit indices (WLSMV and Theta Parameterization used in both models).

Question: I thought, that restricting a model should yield in an equal or worse fit ... Are CFI and RMSEA better because of parsimony or do you think there are mistakes in the syntax?

Data:
Model 1: Configural invariance; factor loadings and intercepts free, latent means fixed in all groups;
Result: Chi2=161.045, df=104;free par.=116;CFI=.975,RMSEA=.051, last Function-value in Tech5 output: 0.18299438D+00.

Model 2: Intercept Invariance; factor loadings and intercepts equal across groups; latent means fixed in one group.
Chi2=170.101, df=115;CFI=.976,RMSEA=.047.
Function-value in Tech5 output: 0.19829076D+00.

Model 3: Strict factorial invariance; factor loadings and intercepts equal across groups; latent means fixed in one group; equal residual variances (fixed to 1 in each group).
Chi2=165.826, df=115;CFI=.978,RMSEA=.045.
Function-value in Tech5 output: 0.25836231D+00.

Latent variances free in all models.

Difftest 1 vs. 2 is not significant:
Chi2=22.294, df=23, p=.50
Difftest 1 vs. 3 is not significant:
Chi2=53.159, df=43, p=.13
 Linda K. Muthen posted on Monday, September 19, 2011 - 11:10 am
With WLSMV, you cannot compare the value of the chi-square statistics. They may not behave as expected. You should interpret only the DIFFTEST results.
 Peter van de Ven posted on Tuesday, November 13, 2012 - 5:49 am
Dear all,

I have a run a CFA to test for strict factorial invariance of a 12 item list where all items have 3 ordinal categories. I have chosen the theta parametrization and used the WLSMV estimation method. Most fit indices point towards adequate fit of the strict factorial invariance model (Chi-Square/df = 1.6, RMSEA = 0.054, CFI = 0.981, TLI = 0.985). Moreover, the model fit is not worse when compared to the partial factorial invariance model where residual variances are not fixed (p = 0.0526 using DIFFTEST), albeit it borderline. My only concern is the large WRMR for the models: WRMR = 3.125 for the strict invariance model and WRMR = 2.813 for the partial invariance model. Ideally these should not exceed 1. Should I worry about these values? All modification indices are below 20 so not big improvements are to be expected by freeing some parameters.
 Linda K. Muthen posted on Tuesday, November 13, 2012 - 8:39 am
You should not worry about WRMR. It is an experimental fit statistic.
 Jamie R. Johnson posted on Friday, August 16, 2013 - 11:00 am
Dear all;

I am trying to run a strict invariance with dichotomous variables and am having some difficulties. It SEEMS like I have everything right, but I'm getting the following error message:
*** ERROR
The following MODEL statements are ignored:
and then it just lists the variables. I've played around and found I do not get the error message if I remove the rotation line from the syntax (cf-quartimax(oblique), I do not get the error message because Mplus gives no output whatsoever. I'm using the theta parameterization, wlsmv estimator but this odd error message; What is going on?

Many thanks.
 Linda K. Muthen posted on Friday, August 16, 2013 - 12:18 pm
Please send the files and your license number to support@statmodel.com.
 Eiko Fried posted on Wednesday, May 06, 2015 - 5:00 am
In addition to configural, metric, and scalar variance, we want to test residual/strict measurement invariance in a ESEM model. Using MLR, this simply means adding statements like this to the syntax:
x1_time1 (a);
x1_time2 (a);

However, we use the WLSMV estimator, and (using the theta parameterization) can't figure out a way this residual invariance model is nested in previous models. Any help would be appreciated. You stated above:
"The models that test this are a model with all residual variances fixed to one in all groups versus a model with residual variances fixed to one in one group and free in the other groups."

But these are two models and not one strict invariance model? And neither of them seems nested in previous models (that do not specificy/estimate residual variances)?
 Bengt O. Muthen posted on Wednesday, May 06, 2015 - 2:46 pm
Because you have to have one group with fixed 1 Theta residual variances, strict invariance is fixing all groups' residual variances at 1. Things are different with categorical outcomes than continuous outcomes; you lose some information.
 Eiko Fried posted on Thursday, May 07, 2015 - 2:31 am
Bengt, what I don't understand is why constraining residuals to 1 in the strict invariance model leads to exactly the same model fit and DF as the strong invariance model.

Strong invariance:
F1-F3 by H1_1-H17_1 (*t1 1);
F4-F6 by H1_2-H17_2 (*t2 1);
H1_1-H17_1 PWITH H1_2-H17_2;
[F1@0];
[F2@0];
[F3@0];
[F4];
[F5];
[F6];
[H1_1$1] (1);
[H1_2$1] (1);
[... and so forth for all thresholds]
chi^2 4299.74 with 532 DF

Strict invariance:
Additionally:
H1_1@1;
H1_2@1;
H2_1@1;
H2_2@1;
[... and so forth for all residuals]
chi^2 4299.74 with 532 DF
 Bengt O. Muthen posted on Thursday, May 07, 2015 - 9:02 am
If you look at the output for your strong invariance run I think you will see that the residual variances are all fixed at 1 (using WLSMV). You need to free them for the time point 2 items.
 Eiko Fried posted on Thursday, May 07, 2015 - 9:25 am
Thanks Bengt. In the input for the strong invariance run in do not specify anything about the residual variances (apart from allowing them to be correlated over time with the pwith command). The output of the strong invariance model contains no information about residual variances at all (only item loadings & thresholds, factor means & variances).

If I rerun the strong invariance model, adding,
H1_1@1;
H1_2@1;
(constraining the residual variance of one item to be the same across time)

I receive exactly the same chi-square and DF as in the model without these. But now the output contains the two residual variances (both at 1).

To me that means that Mplus constrains residuals by default to 1 across time. But that does not make much sense ... what am I missing?
 Bengt O. Muthen posted on Thursday, May 07, 2015 - 1:52 pm
Mplus does not know what you intend. You should explicitly free the residual variances for the items at time 2 in your strong invariance run.
 Eiko Fried posted on Friday, May 08, 2015 - 4:23 am
Bengt, following your suggestion the strong invariance model is not nested anymore in the weak invariance models.

What I am looking for are 4 nested subsequent measurement invariance models, from no invariance to strict invariance. This seems to require the theta parameterization.

See abbreviated syntax for the four models below. Again, the main problem is that the 3rd and 4th models is not nested in the first two.

ANALYSIS:
estimator=WLSMV;
parameterization = theta;


NO INVARIANCE
F1-F3 by H1_1-H17_1 (*t1 1);
F4-F6 by H1_2-H17_2 (*t2 2);
H1_1-H17_1 PWITH H1_2-H17_2;

[F1@0];
[F2@0];
[F3@0];
[F4@0];
[F5@0];
[F6@0];

WEAK INVARIANCE
same except for constrained factor loadings
F1-F3 by H1_1-H17_1 (*t1 1);
F4-F6 by H1_2-H17_2 (*t2 1);

STRONG INVARIANCE
same except latent factor means free at time 2
[F4];
[F5];
[F6];
additionally:
- all thresholds constrained equal over time
- all residual variances @1 time1, free time2 [!!!this leads to a model that is not nested!!!]

STRICT INVARIANCE
same except for:
- all residual variances @1
 Bengt O. Muthen posted on Friday, May 08, 2015 - 8:19 am
With categorical items, invariant factor loadings only (and not invariant thresholds) cannot be identified together with free residual variances (for time 2) unless your items have more than 2 categories. So in your binary case when you go from your weak invariance model, which doesn't have any free residual variances I assume, to your strong invariance model you add residual variances, causing the non-nested problem. We show how we do configural, metric, and scalar testing for categorical items in our 7.1 Language Addendum on our website, in the section Convenience Features. This relates to Millsap's work which we refer to.

So, the binary case does not fall neatly into the testing sequence of continuous items that you are attempting.
 Eiko Fried posted on Friday, May 08, 2015 - 4:05 pm
Bengt, thank you so much for your insights.

We have the case of ordered-categorical items with 4 categories (and not dichotomous items). How does one build a nested strict invariance model in that specific case?

I read the language addendum information, but it doesn't feature a residual variance model which is required for measurement invariance (at least according to Meredith's highly influential paper).
 Bengt O. Muthen posted on Sunday, May 10, 2015 - 3:12 pm
Use the approach discussed in the language addendum for configural, metric, and scalar invariance with polytomous items. Then add to the scalar model the model where the residual variances are equal, which is the same as them being fixed at one in all instances.
 Eiko Fried posted on Monday, May 11, 2015 - 10:02 am
Thanks Bengt. Following the procedure described, in the configural model, theta parameterization, I should——in addition to fixing all factor means to 0 and all residual variances to 1——set the factor variance of one or all factors (it's not clear for me from reading it) to 1. In both cases I receive the error for my ESEM models:

The variance of EFA factors in the reference block cannot be modified.

So how do I do that in the ESEM case?
 Bengt O. Muthen posted on Monday, May 11, 2015 - 3:04 pm
Because ESEM behaves like EFA, it already fixes the factor variances at 1.
 Eiko Fried posted on Tuesday, May 12, 2015 - 4:42 am
Thanks. All 4 models converge, but all give 2 PSI warnings. With increasing constraints, the latent variable covariances more and more go above 1. In the strict invariance model, about 20% of all covariances of the 6 factors are above 1 (the covariance of one factor with itself is >4).

What is the standard way to deal with this problem?

Also, chi-square increases from configural to strong invariance, but then drops substantially in the strict invariance model. Can I ignore that because chi-square is not really interpretable with WLSMV, or is that reason to be concerned? (DFs all increase, and all DIFFTESTs are significant)
 Bengt O. Muthen posted on Tuesday, May 12, 2015 - 7:36 am
Q1. Use fewer factors.

Q2. With WLSMV the absolute chi-2 size is not the important part, but the Difftest results.
 Eiko Fried posted on Tuesday, May 12, 2015 - 8:35 am
Thank you, extracting 2 factors per group instead of 3 resolves the PSI warnings.

However, these models fit worse, and parallel analysis and scree plot suggest extracting at least 3 factors in each group.

Is there a well-established argument that can be used to use less factors in such cases?
 Bengt O. Muthen posted on Tuesday, May 12, 2015 - 5:57 pm
No. It is more a matter of trying to modify the measurement instrument, e.g. adding items. Not all measurement instruments are well-fitted by a factor model, no matter how many factors you choose. I assume you are using EFA/ESEM so that the reason for high factor correlations isn't the usual CFA reason.
 Eiko Fried posted on Wednesday, May 13, 2015 - 2:51 am
Yes we're using EFA/ESEM. We're analyzing 4 different rating scales for depression (16-30 items), and since 1-7 factor solutions have been reported for each in the last 40 yrs, choosing an a priori CFA structure is difficult (even when doing it data-driven, over 50% of all items have high cross loadings).

The scales are not really "psychometric" measures; clinicians made up items, and for some reason the instruments are still used as gold standard half a century later.

So I guess it is not surprising that no proper factor structures emerge. Thanks for the help!
 Bengt O. Muthen posted on Wednesday, May 13, 2015 - 10:25 am
Right. This happens often.
 Jan Stochl posted on Thursday, August 08, 2019 - 2:35 am
Dear Bengt,
I am trying to assess MI of scale (9 polytomous items) across 10 occasions using the same approach as Eiko above in this threat. I have followed the language addendum as you suggested (WLSMV, theta).
All models are fine except metric where I have error message that the model is not identified. I cannot figure out why. Below is the trimmed syntax.

!factor loadings equal across time
f1-f3 BY i1_1-i9_1 (*t1 1);
f4-f6 BY i1_2-i9_2 (*t2 1);
...
f28-f30 BY i1_10-i9_10 (*t10 1);

!factor means fixed at zero at time 1 and free at other timepoints
[f1@0]; [f2@0]; [f3@0];
[f4]; ... [f30];

!factor variances free across time.
f1-f30;

!residuals correlate across time
i1_1-i9_1 PWITH i1_2-i9_2;
...
i1_9-i9_9 PWITH i1_10-i9_10;

!first 2 thresholds of each item equal across time.
[i1_1$1-i1_1$2](c1-c2);
...
[i9_1$1-i9_1$2](c17-c18);

[i1_2$1-i1_2$2](c1-c2);
...
[i9_2$1-i9_2$2](c17-c18);

...and so on for remaining 8 timepoints.

!residual variances fixed at time 1 and free at remaining timepoints
i1_1-i9_1@1;
i1_2-i9_2;
...
i1_9-i9_9;

Thank you for your help!
 Bengt O. Muthen posted on Thursday, August 08, 2019 - 10:05 am
I think you need to fix the factor variance at time 1. See the UG ex52.6 text.
 Jan Stochl posted on Monday, August 12, 2019 - 2:28 am
Very helpful, thank you Bengt! Jan
Back to top
Add Your Message Here
Post:
Username: Posting Information:
This is a private posting area. Only registered users and moderators may post messages here.
Password:
Options: Enable HTML code in message
Automatically activate URLs in message
Action: