Diff test single and 2nd order factor
Message/Author
 Peter Ji posted on Tuesday, September 16, 2008 - 6:17 pm
Hello,

I tried using the Diff test between two models. Both fit, actually the fit stats are almost the same, and I want to see if one model fits better than the other. I have 5 waves of data.

A model with 8 factors correlated did not fit across all 5 waves so we decided to drop that model from further analysis.

When I ran the diff test, the message I got was that the models were not nested. I believe that Model 2 is not nested within Model 1. Because of this the diff test will not run. Is my belief correct? If so is there any way to run a diff test on models that are not nested; anyway to run a comparison between two models that fit to see if one model gives an advantage over another model?

Thank you very much
 Peter Ji posted on Tuesday, September 16, 2008 - 6:32 pm

Comparing CFA models 10-04-06 1:28 am Doveh Etti

the conclusion was that there was no method to compare non-nested models. Is that still true?

thanks.
 Linda K. Muthen posted on Wednesday, September 17, 2008 - 10:47 am
It may be that Model 1 is nested in Model 2. You can try reversing the order of the analysis to see.

To compare non-nested models, you can use BIC for models with the same set of observed variables. However, with weighted least squares estimation, BIC is not available.
 EFried posted on Thursday, June 28, 2012 - 6:01 pm
I am trying to determine:

(1) Whether 1 vs. 2 factor solution CFA over 5 measurement points with ordinal variables (WLSMV estimator) fits better.
I'm using standard CFA for the 1 factor solution, and an ESEM model for the 2 factor solution, syntax being
F1_0-F2_0 by x1 x2 x3 x4 ... (*1); !first measurement point
F1_1-F2_1 by x1 x2 x3 x4 ... (*2); !second measurement point
etc.

How do I determine whether 1 or 2 factor solution in this case fits better (assuming CFI & WRSMR imply about the same fit)?
Also, are the eigenvalues extracted in EFA for ordinal variables in MPLUS reliable? I remember there was some discussion about these some years back.

If I understand correctly the scaling correction factor is needed for the CHI^2 testing, but I cannot find it in the output.

(2) Measurement variance of the same factor solution over time (introducing factor loading invariance, thresholds invariance, etc). Do I understand correctly that one uses difftest for it?

Thanks
 Bengt O. Muthen posted on Friday, June 29, 2012 - 8:53 am
You should use Difftest for both (1) and (2).

For your second measurement point, I think you mean to show different x variables.

The eigenvalues are indeed reliable. With WLSMV, they are computed for the sample tetrachoric/polychoric correlations.
 EFried posted on Friday, June 29, 2012 - 3:10 pm
Thank you Bengt!

(1) I'm running the unconstrained 2-factor ESEM first, saving the derivatives, then adding factor invariance and running the 2-factor ESEM again, and trying to run difftest.

THE CHI-SQUARE DIFFERENCE TEST COULD NOT BE COMPUTED BECAUSE THE FILE
CONTAINING INFORMATION ABOUT THE H1 MODEL HAS INSUFFICIENT DATA.

Both ESEM models converge normally. Syntax see in my posting above (you were right that the x also change over time, of course).
The derivates file is 5 mb long and contains ~200.000 lines, which are mostly filled with
0.000000000000000E+000

In the last two cent or so there are some other values.

(2) Comparing 1 to 2 factor models, the 1 Factor Model is the H1 to save derivatives from, the 2 Factor the more constrained H0?

Thank you
 EFried posted on Friday, June 29, 2012 - 3:57 pm

 Linda K. Muthen posted on Sunday, July 01, 2012 - 10:43 am
 EFried posted on Monday, July 02, 2012 - 2:01 pm
I found in another forum post that imputed data cannot be compared that way. It works now.

The last issue I am having is trying to extract Eigenvalues for an ordinal ESEM model with 2 factors and multiple timepoints.

Using EFA doesn't work here because EFA doesn't understand the fact that items are nested in timepoints.

F1.1-F2.1 by x1 y1 z1... (*1); !first measurement point
F1.2-F2.2 by x2 y2 z2... (*2); !second measurement point

Is there a way to do this within the MPLUS framework?
Thank you.
 Linda K. Muthen posted on Tuesday, July 03, 2012 - 10:23 am
Run two separate EFA's.
 Daniel Lee posted on Tuesday, May 03, 2016 - 11:20 am
Hi,

I have conducted an EFA (categorical, WLSVM) on an 18 item scale, and have elected to keep 2 factors. However, the model fit poorly, and by utilizing the modification index, found that 4 items in particular were greatly contributing to misfit. With the removal of the 4 items, the 2-factor solution fit really well.

Now, I would like to, if possible, compare the first model (all 18 items) to the second model (14 items). It is my understanding that these models are not nested, but truly different. Is there a way to analytically show that the second model fits SIGNIFICANTLY better than the first? I tried using the DIFF TEST (for models that use categorical indicators) and came up with this error:

THE CHI-SQUARE DIFFERENCE TEST COULD NOT BE COMPUTED BECAUSE THE H0 MODEL
IS NOT NESTED IN THE H1 MODEL.

I would greatly appreciate your guidance! Thank you so much!
 Linda K. Muthen posted on Tuesday, May 03, 2016 - 11:42 am
Nested and other model comparisons must at a minimum have the same set of dependent variables.