Message/Author 

Peter Ji posted on Tuesday, September 16, 2008  6:17 pm



Hello, I tried using the Diff test between two models. Both fit, actually the fit stats are almost the same, and I want to see if one model fits better than the other. I have 5 waves of data. Model 1  all variables loading on a single factor Model 2  45 variables each loading on one of 8 factors, 8 factors loading onto a 2nd order factor. A model with 8 factors correlated did not fit across all 5 waves so we decided to drop that model from further analysis. When I ran the diff test, the message I got was that the models were not nested. I believe that Model 2 is not nested within Model 1. Because of this the diff test will not run. Is my belief correct? If so is there any way to run a diff test on models that are not nested; anyway to run a comparison between two models that fit to see if one model gives an advantage over another model? Thank you very much 

Peter Ji posted on Tuesday, September 16, 2008  6:32 pm



I just read this additional post from Comparing CFA models 100406 1:28 am Doveh Etti the conclusion was that there was no method to compare nonnested models. Is that still true? thanks. 


It may be that Model 1 is nested in Model 2. You can try reversing the order of the analysis to see. To compare nonnested models, you can use BIC for models with the same set of observed variables. However, with weighted least squares estimation, BIC is not available. 

EFried posted on Thursday, June 28, 2012  6:01 pm



I am trying to determine: (1) Whether 1 vs. 2 factor solution CFA over 5 measurement points with ordinal variables (WLSMV estimator) fits better. I'm using standard CFA for the 1 factor solution, and an ESEM model for the 2 factor solution, syntax being F1_0F2_0 by x1 x2 x3 x4 ... (*1); !first measurement point F1_1F2_1 by x1 x2 x3 x4 ... (*2); !second measurement point etc. How do I determine whether 1 or 2 factor solution in this case fits better (assuming CFI & WRSMR imply about the same fit)? Also, are the eigenvalues extracted in EFA for ordinal variables in MPLUS reliable? I remember there was some discussion about these some years back. If I understand correctly the scaling correction factor is needed for the CHI^2 testing, but I cannot find it in the output. (2) Measurement variance of the same factor solution over time (introducing factor loading invariance, thresholds invariance, etc). Do I understand correctly that one uses difftest for it? Thanks 


You should use Difftest for both (1) and (2). For your second measurement point, I think you mean to show different x variables. The eigenvalues are indeed reliable. With WLSMV, they are computed for the sample tetrachoric/polychoric correlations. 

EFried posted on Friday, June 29, 2012  3:10 pm



Thank you Bengt! (1) I'm running the unconstrained 2factor ESEM first, saving the derivatives, then adding factor invariance and running the 2factor ESEM again, and trying to run difftest. THE CHISQUARE DIFFERENCE TEST COULD NOT BE COMPUTED BECAUSE THE FILE CONTAINING INFORMATION ABOUT THE H1 MODEL HAS INSUFFICIENT DATA. Both ESEM models converge normally. Syntax see in my posting above (you were right that the x also change over time, of course). The derivates file is 5 mb long and contains ~200.000 lines, which are mostly filled with 0.000000000000000E+000 In the last two cent or so there are some other values. (2) Comparing 1 to 2 factor models, the 1 Factor Model is the H1 to save derivatives from, the 2 Factor the more constrained H0? Thank you 

EFried posted on Friday, June 29, 2012  3:57 pm



Sorry, first sentence should read ... then adding factor *LOADING* invariance ... 


Please send the output showing the problem and your license number to support@statmodel.com. 

EFried posted on Monday, July 02, 2012  2:01 pm



I found in another forum post that imputed data cannot be compared that way. It works now. The last issue I am having is trying to extract Eigenvalues for an ordinal ESEM model with 2 factors and multiple timepoints. Using EFA doesn't work here because EFA doesn't understand the fact that items are nested in timepoints. F1.1F2.1 by x1 y1 z1... (*1); !first measurement point F1.2F2.2 by x2 y2 z2... (*2); !second measurement point … Is there a way to do this within the MPLUS framework? Thank you. 


Run two separate EFA's. 

Daniel Lee posted on Tuesday, May 03, 2016  11:20 am



Hi, I know some updates have been made to mplus since 2008 and wanted to follow up about comparing two nonnested ESEM models: I have conducted an EFA (categorical, WLSVM) on an 18 item scale, and have elected to keep 2 factors. However, the model fit poorly, and by utilizing the modification index, found that 4 items in particular were greatly contributing to misfit. With the removal of the 4 items, the 2factor solution fit really well. Now, I would like to, if possible, compare the first model (all 18 items) to the second model (14 items). It is my understanding that these models are not nested, but truly different. Is there a way to analytically show that the second model fits SIGNIFICANTLY better than the first? I tried using the DIFF TEST (for models that use categorical indicators) and came up with this error: THE CHISQUARE DIFFERENCE TEST COULD NOT BE COMPUTED BECAUSE THE H0 MODEL IS NOT NESTED IN THE H1 MODEL. I would greatly appreciate your guidance! Thank you so much! 


Nested and other model comparisons must at a minimum have the same set of dependent variables. 

Back to top 