Message/Author 


Hello, I wish to compare 2, 3, and 4 factor CFA models which are (I believe) nonnested. These are all "standard" CFA models and all share the same exact variable pool (and all variables are ordered binary). However, the models which contain more factors are not created by simply splitting the factors from the models which contain fewer factors. For example, in the two factor model the following is specified: MODEL: f1 BY lintrst decapp decwate incapp incwate earlins midins latins hypins fatigue divar retard agit pleasur lsex conc thgt indeci; f2 BY worthls sinful guilty conf inferio death wdie commit attempt; In the three factor model: MODEL: f1 BY decapp decwate earlins midins latins agit; f2 BY death wdie commit attempt; f3 BY lintrst hypins fatigue divar retard pleasur lsex conc thgt indeci incapp incwate worthls sinful guilty conf inferio; Finally, in the four factor model: MODEL: f1 BY decapp decwate earlins midins latins divar retard agit; f2 BY death wdie commit attempt; f3 BY incapp incwate; f4 BY lintrst hypins fatigue pleasur lsex conc thgt indeci worthls sinful guilty conf inferio; So, my first question is: Are (any of) these models nested? If so, how? Second, if they are not nested (which is my understanding), is there any way I can compare the fit of these models with one another? I tried to use the MLR estimator in order to obtain BIC values for each of these models, but the 4factor model would not estimate (error was something to the effect of: not enough physical memory, over 50,000 integration points required), and the 3factor model probably would have estimated but it would have took days to weeks for the computer to estimate the model. Can I calculate the BIC (or AIC) by hand with the WLSMV estimator? If so, how? If not, how would you compare the fit of these models? Thank you so much, Jim 

BMuthen posted on Thursday, April 14, 2005  12:04 am



I don't believe that these models are nested. You can use BIC. To make the three and four factor models computationally feasible, reduce the number of integration points by saying INTEGRATION = 5; to get an approximate ML solution. You cannot get BIC via WLSMV because you need the loglikelihood value which you can only get through maximum likelihood. Be sure you are using Version 3.12 as there have been significant speed improvements for multiple integration. 


I was so glad to come across this thread as I am dealing with a very similar situation (comparing nonnested 2 and 3factor categorical CFA models). Dr. Muthen, could you please explain how reducing the number of integration points permits the use of ML and BIC? Many Thanks  Jason 


With maximum likelihood and categorical outcomes, numerical integration is required. You can read about numerical integration on pages 330333 of the Mplus User's Guide. Reducing the number of intergration points makes the analysis computationally feasible when there are more than a couple of factors. 

Sarah Olivo posted on Wednesday, April 15, 2009  8:43 am



I am attempting to compare nonnested, secondorder models which include continuous and categorical data. Model specification is listed below. Based on my understanding of your statement above, "Reducing the number of intergration points makes the analysis computationally feasible when there are more than a couple of factors," am I to assume that using numerical integration is not feasible for my purposes since I have either 1 or 2 higher order factors? Also, theoretically it seems that these models should be nested, although using the DIFFTEST function, Mplus tells me they're not. Syntax: MODEL: MDD BY pmdd3 pmdd4 pmdd5; GAD BY pargad1 pargad2 pargad3; SOC BY parsoc1 parsoc2 parsoc3; PANIC BY parpd2 parpd6 parpd7; OCD BY parob2 parob3 parob9; NA BY MDD GAD SOC PANIC OCD; MODEL: MDD BY pmdd3 pmdd4 pmdd5; GAD BY pargad1 pargad2 pargad3; SOC BY parsoc1 parsoc2 parsoc3; PANIC BY parpd2 parpd6 parpd7; OCD BY parob2 parob3 parob9; NA BY MDD GAD SOC PANIC OCD; MDD WITH SOC; 


Please send the two outputs and your license number to support@statmodel.com. 


Hello. I am trying to determine which of several nonnested models is the best fit for my data. The outcome variable is binary (yes/no) so I used the WLSMV estimator. The same variables are being used for all models. How does one go about determining the best fitting model in this situation? 


With weighted least squares, I know of no test statistic that can compare nonnested models. 


Is the above still the case (i.e., no test statistic that can compare nonnested models when using wlsmv)? If so, what is the recommended estimator/test statistic combination when comparing nonnested models estimated on the polychoric correlation matrix? 


This is still true as far as I know. 

Catherine posted on Wednesday, January 25, 2012  10:52 am



I want to compare two nonnested models of categorical indicators.. but i dont know how.. is it even possible? 


If you are using maximum likelihood estimation, you can use BIC. 


Dear Linda, Please could you advise how to compare nonnested models with all continuous variables. I use MLR estimation for the analysis. Both models in comparison are using the same variables but the structural paths are different. In effect, the only difference in these two models is the number of degree of freedom. Thanks. pat 


Dear Linda, Sorry I gave wrong information in the previous message. Please ignore the previous message. The structural models in comparision are not nested. They have the same number of manifest variables, but not the same variables. These are all continuous variables. Structural paths are slightly different in both number and direction. Please could you advise how to compare these two models. Thanks. Pat 


The models cannot be compared. The most you can do is look at fit for each model. 


I have come across a webpage explaining that we can use AIC as a measure to identify if one model is better than the other as long as the models in comparison have the same number of manifest variables, but not necessarily the same variables. If this is invalid, what is the boundary condition that we could use AIC for models comparison? Thanks. 


If you think this is a valid comparison, by all means make it. 


Apology for the miscommunication in the previous message. I didn't make it clear. I don't think the guideline provided on that webpage is valid. I feel that a comparison of structural models with different manifest variables is not a valid comparison although these models have the same number of manifest variables (but I might be wrong). I'd like to have your view on this. Also, I'd like to understand when we can use AIC as the measure for models comparison. Do you know any books or papers that would help me undrestand this. Thanks. 


I believe you need the same set of variables to use BIC or AIC to compare models. This is because without the same variables, they are not in the same metric. See the following FAQ in the website: BIC citations of interest  how big a difference 


Thanks, Linda. 


Dear Linda, I wanted to compare the fit of two sixfactor CFA models with covariates (using WLSMV estimation). One model has all the covariates loading on six factors while in the second model I've removed all nonsignificant correlations (including removing one covariate from the model that was not significantly correlated with any factors). Models are below (note, ethn, chgend, sibgend & oldyng are dichotomous). Can I use AIC to compare the fit, or is this not possible because the second model does not contain the Age variable? If it's not possible to compare with AIC for this reason do you know of any references I can cite to explain why I did not compare the models with a test statistic? Thanks very much Naomi MODEL 1: f1 BY s1 s2 s3; f2 BY s4 s5 s6; f3 BY s7 s8 s9; f4 BY s10 s11 s12; f5 BY s13 s14 s15; f6 BY s16 s17 s18; ethn WITH f1 f2 f3 f4 f5 f6; chgend WITH f1 f2 f3 f4 f5 f6; sibgend WITH f1 f2 f3 f4 f5 f6; oldyng WITH f1 f2 f3 f4 f5 f6; agegap WITH f1 f2 f3 f4 f5 f6; age WITH f1 f2 f3 f4 f5 f6; ses WITH f1 f2 f3 f4 f5 f6; MODEL 2: f1 BY s1 s2 s3; f2 BY s4 s5 s6; f3 BY s7 s8 s9; f4 BY s10 s11 s12; f5 BY s13 s14 s15; f6 BY s16 s17 s18; ethn WITH f1 f2 f4 f5 f6; chgend WITH f1 f3 f4 f5; sibgend WITH f2; oldyng WITH f2 f6; agegap WITH f6; ses WITH f6; 


Models with the same set of dependent variables can be compared using BIC. But WLSMV does not give BIC. Nonnested models using WLSMV cannot be compared. 

Margarita posted on Saturday, June 07, 2014  10:50 am



Dear Dr. Muthen, I hope you are well. I understand that BIC/AIC are not provided when WLSMV is used. However, would it be wrong to do it manually for nonnested models? Thank you 


Yes, the are based on the loglikelihood which is not available for weighted least squares estimation. 

ri ri posted on Tuesday, September 09, 2014  12:17 am



I have viewed the discussion here. I also have categorical DVs and want to compare nonnested models. I used WLSMV to test models. Just for the purpose of comparing models, can I use the MLR so that I can get a BIC or AIC? I want to find a way to compare two models that are not nested with categorical data. 


If your models have the same set of dependent variables, they can be compared using BIC which is available with maximum likelihood estimation. You cannot compare the models using ML and present WLSMV results. 

Mark LaVenia posted on Tuesday, December 16, 2014  11:01 am



I am comparing nonnested models. Can the BICs be compared when using MLR, or would I need to use ML if I wish to compare the BICs. My question stems from knowing that a scaling correction is needed when doing a difference test with MLR. Given that my models are not nested, I'm aware that the difference testing is not an option. However, I'm not 100% clear on whether the BICs can be directly compared when using MLR estimation. Thank you. 


BICs are the same for ML and MLR (only SEs and chi2 differ). No scaling corrections are involved. BIC is suitable for nonnested and nested models. 

Margarita posted on Tuesday, February 23, 2016  12:04 pm



Dear Pr. Muthén, I would like to compare the following 3 models with continuous variables: ModelA: QOLb ON REAP SUPP EA PCL; REAP ON PCL; SUPP ON PCL; EA ON PCL; REAP WITH EA; SUPP WITH EA; MODEL INDIRECT: QOLb IND PCL; ModelB: QOLB ON EA SUPP REAP PCL; EA ON REAP SUPP PCL; REAP ON PCL; SUPP ON PCL; MODEL INDIRECT: QOLB IND PCL; ModelC: QOLB ON EA PCL REAP SUPP; REAP ON PCL; SUPP ON PCL; EA ON PCL; REAP ON EA; SUPP ON EA; MODEL INDIRECT: QOLB IND PCL; I've been told at my viva that I can use chisquare difference to compare these models but I disagree. Am I right to assume that the three models are not nested and the only way to compare them is by using AIC and BIC? Thank you in advance 


You may want to try a general analysis question like this on SEMNET. 


Dear Mplus Team, I’m comparing different factor models of the same construct using CFA with categorical factor indicators in a sample of 700 participants using the default WLSMV estimator. Because some of the factor models are nonnested I want to use BIC in order to be able to compare these models. To obtain BIC I reestimated the models using the MLR estimator with MONTECARLO numerical integation with 5000 integration points. Because one of the models is a 5factor model I expected the model estimation to be very slow. It converged, however, within 30 minutes. Therefore, I’m hesitating to trust the model results and the BICvalue, although Mplus reported that the model estimation terminated normally. As a next step, I compared the parameter estimates as estimated with the MLR estimator to the parameter estimates as estimated with the WLSMV estimator. Although the unstandardized parameter estimates and standard errors differ considerably, the standardized parameter estimates are similar. My question is whether the unstandardized parameter estimates and standard errors as estimated with MLR should be similar to the parameter estimates as estimated with WLSMV and do you think I can trust the BICvalue that was reported by the MLR estimated model in this situation? Thank you very much for your reaction. Best regards, Niels van der Aa 


WLSMV uses probit and MLR uses logit so that explains differences in the unstandardized results. 


Thank you. Do you think I can trust the BIC value and parameter estimates resulting from the MLR estimated model in this situation, especially because the model which I expected to be computionally very demanding converged within 30 minutes? 


With Monte Carlo integration for 5 dimensions the loglikelihood is a bit approximate and therefore also BIC. But big differences between models should be trustworthy. 

Yoosun Chu posted on Thursday, August 10, 2017  8:44 am



Hello, Based on the above discussion, I understand that nonnested models using the WLSMV estimation cannot be compared with each other. From the fit statistics, we can just see whether or not the model fits the data well. Is this correct understanding? Also, I estimated singlelevel CFA using ordinal outcomes and WLSMV estimation. Then, I estimated twolevel CFA using the same indicators and samples. I assume the same structure in between. The fit statistics of both single and twolevel CFA are good. My question is: is there any way to compare the singlelevel CFA and the twolevel CFA? Thanks. 


Q1: Yes Q2: No 


Dear all Thank you for the info in this tread. Can you recommend a reference to use as an argument for not being able to compare nonnested WLSMV models? Best 


No; try SEMNET. 


Thanks! 


I am attempting to compare models estimated with WLSMV via DIFFTEST that I believe are nested, but Mplus says they're not and therefore does not compare them. Is this an error, and if so, is there a way to calculate the difference test by hand? Simplified version of the models: TITLE: H1 VARIABLE: NAMES = y1y4 x1; USEVARIABLES = y1x1 x1sq; CATEGORICAL = y1y4; DEFINE: x1sq = x1*x1; ANALYSIS: ESTIMATOR = WLSMV; MODEL: f1 BY y1y4* (1); f1@1; f1 ON x1 x1sq; SAVEDATA: DIFFTEST = H1.dat; TITLE: H0 VARIABLE: NAMES = y1y4 x1; USEVARIABLES = y1x1 x1sq; CATEGORICAL = y1y4; DEFINE: x1sq = x1*x1; ANALYSIS: ESTIMATOR = WLSMV; DIFFTEST = H1.dat; MODEL: f1 BY y1y4* (1); f1@1; f1 ON x1 x1sq@0; 


Try the new NESTED option to check if they are nested. See the paper on our website: Asparouhov, T. & Muthén, B. (2018). Nesting and equivalence testing for structural equation models. Structural Equation Modeling: A Multidisciplinary Journal. DOI:10.1080/10705511.2018.1513795 (Download scripts). 


Thank you, that is a very useful new option! The nested test in model H1 confirms nestedness, and difftest in H0 now successfully compares the models. Must have been user error somewhere previously. 

Back to top 