Diff test single and 2nd order factor PreviousNext
Mplus Discussion > Confirmatory Factor Analysis >
 Peter Ji posted on Tuesday, September 16, 2008 - 6:17 pm

I tried using the Diff test between two models. Both fit, actually the fit stats are almost the same, and I want to see if one model fits better than the other. I have 5 waves of data.

Model 1 - all variables loading on a single factor
Model 2 - 4-5 variables each loading on one of 8 factors, 8 factors loading onto a 2nd order factor.

A model with 8 factors correlated did not fit across all 5 waves so we decided to drop that model from further analysis.

When I ran the diff test, the message I got was that the models were not nested. I believe that Model 2 is not nested within Model 1. Because of this the diff test will not run. Is my belief correct? If so is there any way to run a diff test on models that are not nested; anyway to run a comparison between two models that fit to see if one model gives an advantage over another model?

Thank you very much
 Peter Ji posted on Tuesday, September 16, 2008 - 6:32 pm
I just read this additional post from

Comparing CFA models 10-04-06 1:28 am Doveh Etti

the conclusion was that there was no method to compare non-nested models. Is that still true?

 Linda K. Muthen posted on Wednesday, September 17, 2008 - 10:47 am
It may be that Model 1 is nested in Model 2. You can try reversing the order of the analysis to see.

To compare non-nested models, you can use BIC for models with the same set of observed variables. However, with weighted least squares estimation, BIC is not available.
 EFried posted on Thursday, June 28, 2012 - 6:01 pm
I am trying to determine:

(1) Whether 1 vs. 2 factor solution CFA over 5 measurement points with ordinal variables (WLSMV estimator) fits better.
I'm using standard CFA for the 1 factor solution, and an ESEM model for the 2 factor solution, syntax being
F1_0-F2_0 by x1 x2 x3 x4 ... (*1); !first measurement point
F1_1-F2_1 by x1 x2 x3 x4 ... (*2); !second measurement point

How do I determine whether 1 or 2 factor solution in this case fits better (assuming CFI & WRSMR imply about the same fit)?
Also, are the eigenvalues extracted in EFA for ordinal variables in MPLUS reliable? I remember there was some discussion about these some years back.

If I understand correctly the scaling correction factor is needed for the CHI^2 testing, but I cannot find it in the output.

(2) Measurement variance of the same factor solution over time (introducing factor loading invariance, thresholds invariance, etc). Do I understand correctly that one uses difftest for it?

 Bengt O. Muthen posted on Friday, June 29, 2012 - 8:53 am
You should use Difftest for both (1) and (2).

For your second measurement point, I think you mean to show different x variables.

The eigenvalues are indeed reliable. With WLSMV, they are computed for the sample tetrachoric/polychoric correlations.
 EFried posted on Friday, June 29, 2012 - 3:10 pm
Thank you Bengt!

(1) I'm running the unconstrained 2-factor ESEM first, saving the derivatives, then adding factor invariance and running the 2-factor ESEM again, and trying to run difftest.


Both ESEM models converge normally. Syntax see in my posting above (you were right that the x also change over time, of course).
The derivates file is 5 mb long and contains ~200.000 lines, which are mostly filled with

In the last two cent or so there are some other values.

(2) Comparing 1 to 2 factor models, the 1 Factor Model is the H1 to save derivatives from, the 2 Factor the more constrained H0?

Thank you
 EFried posted on Friday, June 29, 2012 - 3:57 pm
Sorry, first sentence should read

... then adding factor *LOADING* invariance ...
 Linda K. Muthen posted on Sunday, July 01, 2012 - 10:43 am
Please send the output showing the problem and your license number to support@statmodel.com.
 EFried posted on Monday, July 02, 2012 - 2:01 pm
I found in another forum post that imputed data cannot be compared that way. It works now.

The last issue I am having is trying to extract Eigenvalues for an ordinal ESEM model with 2 factors and multiple timepoints.

Using EFA doesn't work here because EFA doesn't understand the fact that items are nested in timepoints.

F1.1-F2.1 by x1 y1 z1... (*1); !first measurement point
F1.2-F2.2 by x2 y2 z2... (*2); !second measurement point

Is there a way to do this within the MPLUS framework?
Thank you.
 Linda K. Muthen posted on Tuesday, July 03, 2012 - 10:23 am
Run two separate EFA's.
 Daniel Lee posted on Tuesday, May 03, 2016 - 11:20 am

I know some updates have been made to mplus since 2008 and wanted to follow up about comparing two non-nested ESEM models:

I have conducted an EFA (categorical, WLSVM) on an 18 item scale, and have elected to keep 2 factors. However, the model fit poorly, and by utilizing the modification index, found that 4 items in particular were greatly contributing to misfit. With the removal of the 4 items, the 2-factor solution fit really well.

Now, I would like to, if possible, compare the first model (all 18 items) to the second model (14 items). It is my understanding that these models are not nested, but truly different. Is there a way to analytically show that the second model fits SIGNIFICANTLY better than the first? I tried using the DIFF TEST (for models that use categorical indicators) and came up with this error:


I would greatly appreciate your guidance! Thank you so much!
 Linda K. Muthen posted on Tuesday, May 03, 2016 - 11:42 am
Nested and other model comparisons must at a minimum have the same set of dependent variables.
Back to top
Add Your Message Here
Username: Posting Information:
This is a private posting area. Only registered users and moderators may post messages here.
Options: Enable HTML code in message
Automatically activate URLs in message