I wish to compare 2, 3, and 4 factor CFA models which are (I believe) nonnested. These are all "standard" CFA models and all share the same exact variable pool (and all variables are ordered binary). However, the models which contain more factors are not created by simply splitting the factors from the models which contain fewer factors. For example, in the two factor model the following is specified:
MODEL: f1 BY lintrst decapp decwate incapp incwate earlins midins latins hypins fatigue divar retard agit pleasur lsex conc thgt indeci; f2 BY worthls sinful guilty conf inferio death wdie commit attempt;
In the three factor model:
MODEL: f1 BY decapp decwate earlins midins latins agit; f2 BY death wdie commit attempt; f3 BY lintrst hypins fatigue divar retard pleasur lsex conc thgt indeci incapp incwate worthls sinful guilty conf inferio;
Finally, in the four factor model: MODEL: f1 BY decapp decwate earlins midins latins divar retard agit; f2 BY death wdie commit attempt; f3 BY incapp incwate; f4 BY lintrst hypins fatigue pleasur lsex conc thgt indeci worthls sinful guilty conf inferio;
So, my first question is: Are (any of) these models nested? If so, how?
Second, if they are not nested (which is my understanding), is there any way I can compare the fit of these models with one another? I tried to use the MLR estimator in order to obtain BIC values for each of these models, but the 4-factor model would not estimate (error was something to the effect of: not enough physical memory, over 50,000 integration points required), and the 3-factor model probably would have estimated but it would have took days to weeks for the computer to estimate the model.
Can I calculate the BIC (or AIC) by hand with the WLSMV estimator? If so, how? If not, how would you compare the fit of these models?
Thank you so much, Jim
BMuthen posted on Thursday, April 14, 2005 - 12:04 am
I don't believe that these models are nested. You can use BIC. To make the three and four factor models computationally feasible, reduce the number of integration points by saying INTEGRATION = 5; to get an approximate ML solution. You cannot get BIC via WLSMV because you need the loglikelihood value which you can only get through maximum likelihood. Be sure you are using Version 3.12 as there have been significant speed improvements for multiple integration.
I was so glad to come across this thread as I am dealing with a very similar situation (comparing nonnested 2- and 3-factor categorical CFA models). Dr. Muthen, could you please explain how reducing the number of integration points permits the use of ML and BIC?
With maximum likelihood and categorical outcomes, numerical integration is required. You can read about numerical integration on pages 330-333 of the Mplus User's Guide. Reducing the number of intergration points makes the analysis computationally feasible when there are more than a couple of factors.
Sarah Olivo posted on Wednesday, April 15, 2009 - 8:43 am
I am attempting to compare non-nested, second-order models which include continuous and categorical data. Model specification is listed below. Based on my understanding of your statement above, "Reducing the number of intergration points makes the analysis computationally feasible when there are more than a couple of factors," am I to assume that using numerical integration is not feasible for my purposes since I have either 1 or 2 higher order factors?
Also, theoretically it seems that these models should be nested, although using the DIFFTEST function, Mplus tells me they're not.
Syntax: MODEL: MDD BY pmdd3 pmdd4 pmdd5; GAD BY pargad1 pargad2 pargad3; SOC BY parsoc1 parsoc2 parsoc3; PANIC BY parpd2 parpd6 parpd7; OCD BY parob2 parob3 parob9; NA BY MDD GAD SOC PANIC OCD;
MODEL: MDD BY pmdd3 pmdd4 pmdd5; GAD BY pargad1 pargad2 pargad3; SOC BY parsoc1 parsoc2 parsoc3; PANIC BY parpd2 parpd6 parpd7; OCD BY parob2 parob3 parob9; NA BY MDD GAD SOC PANIC OCD; MDD WITH SOC;
I am trying to determine which of several non-nested models is the best fit for my data. The outcome variable is binary (yes/no) so I used the WLSMV estimator. The same variables are being used for all models.
How does one go about determining the best fitting model in this situation?
Is the above still the case (i.e., no test statistic that can compare non-nested models when using wlsmv)? If so, what is the recommended estimator/test statistic combination when comparing non-nested models estimated on the polychoric correlation matrix?
Please could you advise how to compare non-nested models with all continuous variables. I use MLR estimation for the analysis. Both models in comparison are using the same variables but the structural paths are different. In effect, the only difference in these two models is the number of degree of freedom. Thanks.
Sorry I gave wrong information in the previous message. Please ignore the previous message.
The structural models in comparision are not nested. They have the same number of manifest variables, but not the same variables. These are all continuous variables. Structural paths are slightly different in both number and direction. Please could you advise how to compare these two models. Thanks.
I have come across a webpage explaining that we can use AIC as a measure to identify if one model is better than the other as long as the models in comparison have the same number of manifest variables, but not necessarily the same variables. If this is invalid, what is the boundary condition that we could use AIC for models comparison? Thanks.
Apology for the miscommunication in the previous message. I didn't make it clear. I don't think the guideline provided on that webpage is valid. I feel that a comparison of structural models with different manifest variables is not a valid comparison although these models have the same number of manifest variables (but I might be wrong). I'd like to have your view on this. Also, I'd like to understand when we can use AIC as the measure for models comparison. Do you know any books or papers that would help me undrestand this. Thanks.
I wanted to compare the fit of two six-factor CFA models with covariates (using WLSMV estimation). One model has all the covariates loading on six factors while in the second model I've removed all non-significant correlations (including removing one covariate from the model that was not significantly correlated with any factors). Models are below (note, ethn, chgend, sibgend & oldyng are dichotomous). Can I use AIC to compare the fit, or is this not possible because the second model does not contain the Age variable? If it's not possible to compare with AIC for this reason do you know of any references I can cite to explain why I did not compare the models with a test statistic?
Thanks very much Naomi
MODEL 1: f1 BY s1 s2 s3; f2 BY s4 s5 s6; f3 BY s7 s8 s9; f4 BY s10 s11 s12; f5 BY s13 s14 s15; f6 BY s16 s17 s18;
ethn WITH f1 f2 f3 f4 f5 f6; chgend WITH f1 f2 f3 f4 f5 f6; sibgend WITH f1 f2 f3 f4 f5 f6; oldyng WITH f1 f2 f3 f4 f5 f6; agegap WITH f1 f2 f3 f4 f5 f6; age WITH f1 f2 f3 f4 f5 f6; ses WITH f1 f2 f3 f4 f5 f6;
MODEL 2: f1 BY s1 s2 s3; f2 BY s4 s5 s6; f3 BY s7 s8 s9; f4 BY s10 s11 s12; f5 BY s13 s14 s15; f6 BY s16 s17 s18;
ethn WITH f1 f2 f4 f5 f6; chgend WITH f1 f3 f4 f5; sibgend WITH f2; oldyng WITH f2 f6; agegap WITH f6; ses WITH f6;
Yes, the are based on the loglikelihood which is not available for weighted least squares estimation.
ri ri posted on Tuesday, September 09, 2014 - 12:17 am
I have viewed the discussion here. I also have categorical DVs and want to compare non-nested models. I used WLSMV to test models. Just for the purpose of comparing models, can I use the MLR so that I can get a BIC or AIC? I want to find a way to compare two models that are not nested with categorical data.
If your models have the same set of dependent variables, they can be compared using BIC which is available with maximum likelihood estimation. You cannot compare the models using ML and present WLSMV results.
Mark LaVenia posted on Tuesday, December 16, 2014 - 11:01 am
I am comparing non-nested models. Can the BICs be compared when using MLR, or would I need to use ML if I wish to compare the BICs. My question stems from knowing that a scaling correction is needed when doing a difference test with MLR. Given that my models are not nested, I'm aware that the difference testing is not an option. However, I'm not 100% clear on whether the BICs can be directly compared when using MLR estimation. Thank you.
BICs are the same for ML and MLR (only SEs and chi-2 differ). No scaling corrections are involved. BIC is suitable for non-nested and nested models.
Margarita posted on Tuesday, February 23, 2016 - 12:04 pm
Dear Pr. Muthén,
I would like to compare the following 3 models with continuous variables:
ModelA: QOLb ON REAP SUPP EA PCL;
REAP ON PCL; SUPP ON PCL; EA ON PCL;
REAP WITH EA; SUPP WITH EA;
QOLb IND PCL;
ModelB: QOLB ON EA SUPP REAP PCL;
EA ON REAP SUPP PCL;
REAP ON PCL; SUPP ON PCL;
QOLB IND PCL;
ModelC: QOLB ON EA PCL REAP SUPP;
REAP ON PCL; SUPP ON PCL; EA ON PCL;
REAP ON EA; SUPP ON EA;
QOLB IND PCL;
I've been told at my viva that I can use chi-square difference to compare these models but I disagree. Am I right to assume that the three models are not nested and the only way to compare them is by using AIC and BIC?
I’m comparing different factor models of the same construct using CFA with categorical factor indicators in a sample of 700 participants using the default WLSMV estimator. Because some of the factor models are non-nested I want to use BIC in order to be able to compare these models. To obtain BIC I re-estimated the models using the MLR estimator with MONTECARLO numerical integation with 5000 integration points.
Because one of the models is a 5-factor model I expected the model estimation to be very slow. It converged, however, within 30 minutes. Therefore, I’m hesitating to trust the model results and the BIC-value, although Mplus reported that the model estimation terminated normally. As a next step, I compared the parameter estimates as estimated with the MLR estimator to the parameter estimates as estimated with the WLSMV estimator. Although the unstandardized parameter estimates and standard errors differ considerably, the standardized parameter estimates are similar. My question is whether the unstandardized parameter estimates and standard errors as estimated with MLR should be similar to the parameter estimates as estimated with WLSMV and do you think I can trust the BIC-value that was reported by the MLR estimated model in this situation? Thank you very much for your reaction.
Thank you. Do you think I can trust the BIC value and parameter estimates resulting from the MLR estimated model in this situation, especially because the model which I expected to be computionally very demanding converged within 30 minutes?
With Monte Carlo integration for 5 dimensions the loglikelihood is a bit approximate and therefore also BIC. But big differences between models should be trustworthy.
Yoosun Chu posted on Thursday, August 10, 2017 - 8:44 am
Hello, Based on the above discussion, I understand that non-nested models using the WLSMV estimation cannot be compared with each other. From the fit statistics, we can just see whether or not the model fits the data well. Is this correct understanding?
Also, I estimated single-level CFA using ordinal outcomes and WLSMV estimation. Then, I estimated two-level CFA using the same indicators and samples. I assume the same structure in between. The fit statistics of both single and two-level CFA are good. My question is: is there any way to compare the single-level CFA and the two-level CFA? Thanks.
I am attempting to compare models estimated with WLSMV via DIFFTEST that I believe are nested, but Mplus says they're not and therefore does not compare them. Is this an error, and if so, is there a way to calculate the difference test by hand?
Simplified version of the models: TITLE: H1 VARIABLE: NAMES = y1-y4 x1; USEVARIABLES = y1-x1 x1sq; CATEGORICAL = y1-y4; DEFINE: x1sq = x1*x1; ANALYSIS: ESTIMATOR = WLSMV; MODEL: f1 BY y1-y4* (1); f1@1; f1 ON x1 x1sq; SAVEDATA: DIFFTEST = H1.dat;
TITLE: H0 VARIABLE: NAMES = y1-y4 x1; USEVARIABLES = y1-x1 x1sq; CATEGORICAL = y1-y4; DEFINE: x1sq = x1*x1; ANALYSIS: ESTIMATOR = WLSMV; DIFFTEST = H1.dat; MODEL: f1 BY y1-y4* (1); f1@1; f1 ON x1 x1sq@0;
Try the new NESTED option to check if they are nested. See the paper on our website:
Asparouhov, T. & Muthén, B. (2018). Nesting and equivalence testing for structural equation models. Structural Equation Modeling: A Multidisciplinary Journal. DOI:10.1080/10705511.2018.1513795 (Download scripts).