Peter Ji posted on Tuesday, September 16, 2008 - 6:17 pm
I tried using the Diff test between two models. Both fit, actually the fit stats are almost the same, and I want to see if one model fits better than the other. I have 5 waves of data.
Model 1 - all variables loading on a single factor Model 2 - 4-5 variables each loading on one of 8 factors, 8 factors loading onto a 2nd order factor.
A model with 8 factors correlated did not fit across all 5 waves so we decided to drop that model from further analysis.
When I ran the diff test, the message I got was that the models were not nested. I believe that Model 2 is not nested within Model 1. Because of this the diff test will not run. Is my belief correct? If so is there any way to run a diff test on models that are not nested; anyway to run a comparison between two models that fit to see if one model gives an advantage over another model?
Thank you very much
Peter Ji posted on Tuesday, September 16, 2008 - 6:32 pm
I just read this additional post from
Comparing CFA models 10-04-06 1:28 am Doveh Etti
the conclusion was that there was no method to compare non-nested models. Is that still true?
It may be that Model 1 is nested in Model 2. You can try reversing the order of the analysis to see.
To compare non-nested models, you can use BIC for models with the same set of observed variables. However, with weighted least squares estimation, BIC is not available.
EFried posted on Thursday, June 28, 2012 - 6:01 pm
I am trying to determine:
(1) Whether 1 vs. 2 factor solution CFA over 5 measurement points with ordinal variables (WLSMV estimator) fits better. I'm using standard CFA for the 1 factor solution, and an ESEM model for the 2 factor solution, syntax being F1_0-F2_0 by x1 x2 x3 x4 ... (*1); !first measurement point F1_1-F2_1 by x1 x2 x3 x4 ... (*2); !second measurement point etc.
How do I determine whether 1 or 2 factor solution in this case fits better (assuming CFI & WRSMR imply about the same fit)? Also, are the eigenvalues extracted in EFA for ordinal variables in MPLUS reliable? I remember there was some discussion about these some years back.
If I understand correctly the scaling correction factor is needed for the CHI^2 testing, but I cannot find it in the output.
(2) Measurement variance of the same factor solution over time (introducing factor loading invariance, thresholds invariance, etc). Do I understand correctly that one uses difftest for it?
(1) I'm running the unconstrained 2-factor ESEM first, saving the derivatives, then adding factor invariance and running the 2-factor ESEM again, and trying to run difftest.
THE CHI-SQUARE DIFFERENCE TEST COULD NOT BE COMPUTED BECAUSE THE FILE CONTAINING INFORMATION ABOUT THE H1 MODEL HAS INSUFFICIENT DATA.
Both ESEM models converge normally. Syntax see in my posting above (you were right that the x also change over time, of course). The derivates file is 5 mb long and contains ~200.000 lines, which are mostly filled with 0.000000000000000E+000
In the last two cent or so there are some other values.
(2) Comparing 1 to 2 factor models, the 1 Factor Model is the H1 to save derivatives from, the 2 Factor the more constrained H0?
Daniel Lee posted on Tuesday, May 03, 2016 - 11:20 am
I know some updates have been made to mplus since 2008 and wanted to follow up about comparing two non-nested ESEM models:
I have conducted an EFA (categorical, WLSVM) on an 18 item scale, and have elected to keep 2 factors. However, the model fit poorly, and by utilizing the modification index, found that 4 items in particular were greatly contributing to misfit. With the removal of the 4 items, the 2-factor solution fit really well.
Now, I would like to, if possible, compare the first model (all 18 items) to the second model (14 items). It is my understanding that these models are not nested, but truly different. Is there a way to analytically show that the second model fits SIGNIFICANTLY better than the first? I tried using the DIFF TEST (for models that use categorical indicators) and came up with this error:
THE CHI-SQUARE DIFFERENCE TEST COULD NOT BE COMPUTED BECAUSE THE H0 MODEL IS NOT NESTED IN THE H1 MODEL.
I would greatly appreciate your guidance! Thank you so much!