Comparing non-nested models
Message/Author
 Jim Prisciandaro posted on Tuesday, April 12, 2005 - 1:54 pm
Hello,

I wish to compare 2, 3, and 4 factor CFA models which are (I believe) nonnested. These are all "standard" CFA models and all share the same exact variable pool (and all variables are ordered binary). However, the models which contain more factors are not created by simply splitting the factors from the models which contain fewer factors. For example, in the two factor model the following is specified:

MODEL: f1 BY lintrst decapp decwate incapp incwate earlins midins latins hypins fatigue divar retard agit pleasur lsex conc thgt indeci;
f2 BY worthls sinful guilty conf inferio death wdie commit attempt;

In the three factor model:

MODEL: f1 BY decapp decwate earlins midins latins agit;
f2 BY death wdie commit attempt;
f3 BY lintrst hypins fatigue divar retard pleasur lsex conc thgt indeci incapp incwate worthls sinful guilty conf inferio;

Finally, in the four factor model:
MODEL: f1 BY decapp decwate earlins midins latins divar retard agit;
f2 BY death wdie commit attempt;
f3 BY incapp incwate;
f4 BY lintrst hypins fatigue pleasur lsex conc thgt indeci worthls sinful guilty conf inferio;

So, my first question is: Are (any of) these models nested? If so, how?

Second, if they are not nested (which is my understanding), is there any way I can compare the fit of these models with one another? I tried to use the MLR estimator in order to obtain BIC values for each of these models, but the 4-factor model would not estimate (error was something to the effect of: not enough physical memory, over 50,000 integration points required), and the 3-factor model probably would have estimated but it would have took days to weeks for the computer to estimate the model.

Can I calculate the BIC (or AIC) by hand with the WLSMV estimator? If so, how? If not, how would you compare the fit of these models?

Thank you so much,
Jim
 BMuthen posted on Thursday, April 14, 2005 - 12:04 am
I don't believe that these models are nested. You can use BIC. To make the three and four factor models computationally feasible, reduce the number of integration points by saying INTEGRATION = 5; to get an approximate ML solution. You cannot get BIC via WLSMV because you need the loglikelihood value which you can only get through maximum likelihood. Be sure you are using Version 3.12 as there have been significant speed improvements for multiple integration.
 Jason Fogler posted on Friday, June 02, 2006 - 2:19 pm
I was so glad to come across this thread as I am dealing with a very similar situation (comparing nonnested 2- and 3-factor categorical CFA models). Dr. Muthen, could you please explain how reducing the number of integration points permits the use of ML and BIC?

Many Thanks -- Jason
 Linda K. Muthen posted on Friday, June 02, 2006 - 3:06 pm
With maximum likelihood and categorical outcomes, numerical integration is required. You can read about numerical integration on pages 330-333 of the Mplus User's Guide. Reducing the number of intergration points makes the analysis computationally feasible when there are more than a couple of factors.
 Sarah Olivo posted on Wednesday, April 15, 2009 - 8:43 am
I am attempting to compare non-nested, second-order models which include continuous and categorical data. Model specification is listed below. Based on my understanding of your statement above, "Reducing the number of intergration points makes the analysis computationally feasible when there are more than a couple of factors," am I to assume that using numerical integration is not feasible for my purposes since I have either 1 or 2 higher order factors?

Also, theoretically it seems that these models should be nested, although using the DIFFTEST function, Mplus tells me they're not.

Syntax:
MODEL: MDD BY pmdd3 pmdd4 pmdd5;
SOC BY parsoc1 parsoc2 parsoc3;
PANIC BY parpd2 parpd6 parpd7;
OCD BY parob2 parob3 parob9;
NA BY MDD GAD SOC PANIC OCD;

MODEL: MDD BY pmdd3 pmdd4 pmdd5;
SOC BY parsoc1 parsoc2 parsoc3;
PANIC BY parpd2 parpd6 parpd7;
OCD BY parob2 parob3 parob9;
NA BY MDD GAD SOC PANIC OCD;
MDD WITH SOC;
 Linda K. Muthen posted on Friday, April 17, 2009 - 10:43 am
 Michael S. Businelle, Ph.D. posted on Monday, February 15, 2010 - 2:16 pm
Hello.

I am trying to determine which of several non-nested models is the best fit for my data. The outcome variable is binary (yes/no) so I used the WLSMV estimator. The same variables are being used for all models.

How does one go about determining the best fitting model in this situation?
 Linda K. Muthen posted on Monday, February 15, 2010 - 3:39 pm
With weighted least squares, I know of no test statistic that can compare non-nested models.
 Jamie Marincic posted on Monday, April 11, 2011 - 2:16 pm
Is the above still the case (i.e., no test statistic that can compare non-nested models when using wlsmv)? If so, what is the recommended estimator/test statistic combination when comparing non-nested models estimated on the polychoric correlation matrix?
 Linda K. Muthen posted on Monday, April 11, 2011 - 2:42 pm
This is still true as far as I know.
 Catherine posted on Wednesday, January 25, 2012 - 10:52 am
I want to compare two non-nested models of categorical indicators.. but i dont know how.. is it even possible?
 Linda K. Muthen posted on Wednesday, January 25, 2012 - 11:36 am
If you are using maximum likelihood estimation, you can use BIC.
 Patchara Popaitoon posted on Saturday, May 05, 2012 - 12:59 am
Dear Linda,

Please could you advise how to compare non-nested models with all continuous variables. I use MLR estimation for the analysis. Both models in comparison are using the same variables but the structural paths are different. In effect, the only difference in these two models is the number of degree of freedom. Thanks.

pat
 Patchara Popaitoon posted on Saturday, May 05, 2012 - 1:15 am
Dear Linda,

Sorry I gave wrong information in the previous message. Please ignore the previous message.

The structural models in comparision are not nested. They have the same number of manifest variables, but not the same variables. These are all continuous variables. Structural paths are slightly different in both number and direction. Please could you advise how to compare these two models. Thanks.

Pat
 Linda K. Muthen posted on Saturday, May 05, 2012 - 11:54 am
The models cannot be compared. The most you can do is look at fit for each model.
 Patchara Popaitoon posted on Saturday, May 05, 2012 - 12:27 pm
I have come across a webpage explaining that we can use AIC as a measure to identify if one model is better than the other as long as the models in comparison have the same number of manifest variables, but not necessarily the same variables. If this is invalid, what is the boundary condition that we could use AIC for models comparison? Thanks.
 Linda K. Muthen posted on Saturday, May 05, 2012 - 3:02 pm
If you think this is a valid comparison, by all means make it.
 Patchara Popaitoon posted on Saturday, May 05, 2012 - 11:48 pm
Apology for the miscommunication in the previous message. I didn't make it clear. I don't think the guideline provided on that webpage is valid. I feel that a comparison of structural models with different manifest variables is not a valid comparison although these models have the same number of manifest variables (but I might be wrong). I'd like to have your view on this. Also, I'd like to understand when we can use AIC as the measure for models comparison. Do you know any books or papers that would help me undrestand this. Thanks.
 Linda K. Muthen posted on Sunday, May 06, 2012 - 5:20 pm
I believe you need the same set of variables to use BIC or AIC to compare models. This is because without the same variables, they are not in the same metric. See the following FAQ in the website:

BIC citations of interest - how big a difference
 Patchara Popaitoon posted on Sunday, May 06, 2012 - 11:10 pm
Thanks, Linda.
 Naomi White posted on Friday, May 30, 2014 - 6:11 am
Dear Linda,

I wanted to compare the fit of two six-factor CFA models with covariates (using WLSMV estimation). One model has all the covariates loading on six factors while in the second model I've removed all non-significant correlations (including removing one covariate from the model that was not significantly correlated with any factors). Models are below (note, ethn, chgend, sibgend & oldyng are dichotomous). Can I use AIC to compare the fit, or is this not possible because the second model does not contain the Age variable? If it's not possible to compare with AIC for this reason do you know of any references I can cite to explain why I did not compare the models with a test statistic?

Thanks very much
Naomi

MODEL 1:
f1 BY s1 s2 s3;
f2 BY s4 s5 s6;
f3 BY s7 s8 s9;
f4 BY s10 s11 s12;
f5 BY s13 s14 s15;
f6 BY s16 s17 s18;

ethn WITH f1 f2 f3 f4 f5 f6;
chgend WITH f1 f2 f3 f4 f5 f6;
sibgend WITH f1 f2 f3 f4 f5 f6;
oldyng WITH f1 f2 f3 f4 f5 f6;
agegap WITH f1 f2 f3 f4 f5 f6;
age WITH f1 f2 f3 f4 f5 f6;
ses WITH f1 f2 f3 f4 f5 f6;

MODEL 2:
f1 BY s1 s2 s3;
f2 BY s4 s5 s6;
f3 BY s7 s8 s9;
f4 BY s10 s11 s12;
f5 BY s13 s14 s15;
f6 BY s16 s17 s18;

ethn WITH f1 f2 f4 f5 f6;
chgend WITH f1 f3 f4 f5;
sibgend WITH f2;
oldyng WITH f2 f6;
agegap WITH f6;
ses WITH f6;
 Linda K. Muthen posted on Friday, May 30, 2014 - 1:35 pm
Models with the same set of dependent variables can be compared using BIC. But WLSMV does not give BIC. Non-nested models using WLSMV cannot be compared.
 Margarita  posted on Saturday, June 07, 2014 - 10:50 am
Dear Dr. Muthen,

I hope you are well. I understand that BIC/AIC are not provided when WLSMV is used. However, would it be wrong to do it manually for non-nested models?

Thank you
 Linda K. Muthen posted on Saturday, June 07, 2014 - 5:33 pm
Yes, the are based on the loglikelihood which is not available for weighted least squares estimation.
 ri ri  posted on Tuesday, September 09, 2014 - 12:17 am
I have viewed the discussion here. I also have categorical DVs and want to compare non-nested models. I used WLSMV to test models. Just for the purpose of comparing models, can I use the MLR so that I can get a BIC or AIC? I want to find a way to compare two models that are not nested with categorical data.
 Linda K. Muthen posted on Tuesday, September 09, 2014 - 5:33 am
If your models have the same set of dependent variables, they can be compared using BIC which is available with maximum likelihood estimation. You cannot compare the models using ML and present WLSMV results.
 Mark LaVenia posted on Tuesday, December 16, 2014 - 11:01 am
I am comparing non-nested models. Can the BICs be compared when using MLR, or would I need to use ML if I wish to compare the BICs. My question stems from knowing that a scaling correction is needed when doing a difference test with MLR. Given that my models are not nested, I'm aware that the difference testing is not an option. However, I'm not 100% clear on whether the BICs can be directly compared when using MLR estimation. Thank you.
 Bengt O. Muthen posted on Tuesday, December 16, 2014 - 6:34 pm
BICs are the same for ML and MLR (only SEs and chi-2 differ). No scaling corrections are involved. BIC is suitable for non-nested and nested models.
 Margarita  posted on Tuesday, February 23, 2016 - 12:04 pm
Dear Pr. Muthén,

I would like to compare the following 3 models with continuous variables:

ModelA:
QOLb ON REAP SUPP EA PCL;

REAP ON PCL;
SUPP ON PCL;
EA ON PCL;

REAP WITH EA;
SUPP WITH EA;

MODEL INDIRECT:

QOLb IND PCL;

ModelB:
QOLB ON EA SUPP REAP PCL;

EA ON REAP SUPP PCL;

REAP ON PCL;
SUPP ON PCL;

MODEL INDIRECT:

QOLB IND PCL;

ModelC:
QOLB ON EA PCL REAP SUPP;

REAP ON PCL;
SUPP ON PCL;
EA ON PCL;

REAP ON EA;
SUPP ON EA;

MODEL INDIRECT:

QOLB IND PCL;

I've been told at my viva that I can use chi-square difference to compare these models but I disagree. Am I right to assume that the three models are not nested and the only way to compare them is by using AIC and BIC?

 Bengt O. Muthen posted on Wednesday, February 24, 2016 - 4:40 pm
You may want to try a general analysis question like this on SEMNET.
 Niels van der Aa posted on Monday, May 22, 2017 - 1:22 am
Dear Mplus Team,

I’m comparing different factor models of the same construct using CFA with categorical factor indicators in a sample of 700 participants using the default WLSMV estimator. Because some of the factor models are non-nested I want to use BIC in order to be able to compare these models. To obtain BIC I re-estimated the models using the MLR estimator with MONTECARLO numerical integation with 5000 integration points.

Because one of the models is a 5-factor model I expected the model estimation to be very slow. It converged, however, within 30 minutes. Therefore, I’m hesitating to trust the model results and the BIC-value, although Mplus reported that the model estimation terminated normally. As a next step, I compared the parameter estimates as estimated with the MLR estimator to the parameter estimates as estimated with the WLSMV estimator. Although the unstandardized parameter estimates and standard errors differ considerably, the standardized parameter estimates are similar. My question is whether the unstandardized parameter estimates and standard errors as estimated with MLR should be similar to the parameter estimates as estimated with WLSMV and do you think I can trust the BIC-value that was reported by the MLR estimated model in this situation? Thank you very much for your reaction.

Best regards,
Niels van der Aa
 Bengt O. Muthen posted on Monday, May 22, 2017 - 2:16 pm
WLSMV uses probit and MLR uses logit so that explains differences in the unstandardized results.
 Niels van der Aa posted on Tuesday, May 23, 2017 - 7:22 am
Thank you. Do you think I can trust the BIC value and parameter estimates resulting from the MLR estimated model in this situation, especially because the model which I expected to be computionally very demanding converged within 30 minutes?
 Bengt O. Muthen posted on Tuesday, May 23, 2017 - 5:52 pm
With Monte Carlo integration for 5 dimensions the loglikelihood is a bit approximate and therefore also BIC. But big differences between models should be trustworthy.
 Yoosun Chu posted on Thursday, August 10, 2017 - 8:44 am
Hello,
Based on the above discussion, I understand that non-nested models using the WLSMV estimation cannot be compared with each other. From the fit statistics, we can just see whether or not the model fits the data well. Is this correct understanding?

Also, I estimated single-level CFA using ordinal outcomes and WLSMV estimation. Then, I estimated two-level CFA using the same indicators and samples. I assume the same structure in between. The fit statistics of both single and two-level CFA are good. My question is: is there any way to compare the single-level CFA and the two-level CFA? Thanks.
 Bengt O. Muthen posted on Thursday, August 10, 2017 - 4:39 pm
Q1: Yes

Q2: No
 David Højgaard posted on Thursday, September 21, 2017 - 2:59 am
Dear all

Thank you for the info in this tread.
Can you recommend a reference to use as an argument for not being able to compare non-nested WLSMV models?

Best
 Bengt O. Muthen posted on Thursday, September 21, 2017 - 3:58 pm
No; try SEMNET.
 David Højgaard posted on Friday, September 22, 2017 - 12:31 am
Thanks!