Comparing non-nested models PreviousNext
Mplus Discussion > Categorical Data Modeling >
Message/Author
 Jim Prisciandaro posted on Tuesday, April 12, 2005 - 1:54 pm
Hello,

I wish to compare 2, 3, and 4 factor CFA models which are (I believe) nonnested. These are all "standard" CFA models and all share the same exact variable pool (and all variables are ordered binary). However, the models which contain more factors are not created by simply splitting the factors from the models which contain fewer factors. For example, in the two factor model the following is specified:

MODEL: f1 BY lintrst decapp decwate incapp incwate earlins midins latins hypins fatigue divar retard agit pleasur lsex conc thgt indeci;
f2 BY worthls sinful guilty conf inferio death wdie commit attempt;

In the three factor model:

MODEL: f1 BY decapp decwate earlins midins latins agit;
f2 BY death wdie commit attempt;
f3 BY lintrst hypins fatigue divar retard pleasur lsex conc thgt indeci incapp incwate worthls sinful guilty conf inferio;

Finally, in the four factor model:
MODEL: f1 BY decapp decwate earlins midins latins divar retard agit;
f2 BY death wdie commit attempt;
f3 BY incapp incwate;
f4 BY lintrst hypins fatigue pleasur lsex conc thgt indeci worthls sinful guilty conf inferio;

So, my first question is: Are (any of) these models nested? If so, how?

Second, if they are not nested (which is my understanding), is there any way I can compare the fit of these models with one another? I tried to use the MLR estimator in order to obtain BIC values for each of these models, but the 4-factor model would not estimate (error was something to the effect of: not enough physical memory, over 50,000 integration points required), and the 3-factor model probably would have estimated but it would have took days to weeks for the computer to estimate the model.

Can I calculate the BIC (or AIC) by hand with the WLSMV estimator? If so, how? If not, how would you compare the fit of these models?

Thank you so much,
Jim
 BMuthen posted on Thursday, April 14, 2005 - 12:04 am
I don't believe that these models are nested. You can use BIC. To make the three and four factor models computationally feasible, reduce the number of integration points by saying INTEGRATION = 5; to get an approximate ML solution. You cannot get BIC via WLSMV because you need the loglikelihood value which you can only get through maximum likelihood. Be sure you are using Version 3.12 as there have been significant speed improvements for multiple integration.
 Jason Fogler posted on Friday, June 02, 2006 - 2:19 pm
I was so glad to come across this thread as I am dealing with a very similar situation (comparing nonnested 2- and 3-factor categorical CFA models). Dr. Muthen, could you please explain how reducing the number of integration points permits the use of ML and BIC?

Many Thanks -- Jason
 Linda K. Muthen posted on Friday, June 02, 2006 - 3:06 pm
With maximum likelihood and categorical outcomes, numerical integration is required. You can read about numerical integration on pages 330-333 of the Mplus User's Guide. Reducing the number of intergration points makes the analysis computationally feasible when there are more than a couple of factors.
 Sarah Olivo posted on Wednesday, April 15, 2009 - 8:43 am
I am attempting to compare non-nested, second-order models which include continuous and categorical data. Model specification is listed below. Based on my understanding of your statement above, "Reducing the number of intergration points makes the analysis computationally feasible when there are more than a couple of factors," am I to assume that using numerical integration is not feasible for my purposes since I have either 1 or 2 higher order factors?

Also, theoretically it seems that these models should be nested, although using the DIFFTEST function, Mplus tells me they're not.

Syntax:
MODEL: MDD BY pmdd3 pmdd4 pmdd5;
GAD BY pargad1 pargad2 pargad3;
SOC BY parsoc1 parsoc2 parsoc3;
PANIC BY parpd2 parpd6 parpd7;
OCD BY parob2 parob3 parob9;
NA BY MDD GAD SOC PANIC OCD;

MODEL: MDD BY pmdd3 pmdd4 pmdd5;
GAD BY pargad1 pargad2 pargad3;
SOC BY parsoc1 parsoc2 parsoc3;
PANIC BY parpd2 parpd6 parpd7;
OCD BY parob2 parob3 parob9;
NA BY MDD GAD SOC PANIC OCD;
MDD WITH SOC;
 Linda K. Muthen posted on Friday, April 17, 2009 - 10:43 am
Please send the two outputs and your license number to support@statmodel.com.
 Michael S. Businelle, Ph.D. posted on Monday, February 15, 2010 - 2:16 pm
Hello.

I am trying to determine which of several non-nested models is the best fit for my data. The outcome variable is binary (yes/no) so I used the WLSMV estimator. The same variables are being used for all models.

How does one go about determining the best fitting model in this situation?
 Linda K. Muthen posted on Monday, February 15, 2010 - 3:39 pm
With weighted least squares, I know of no test statistic that can compare non-nested models.
 Jamie Marincic posted on Monday, April 11, 2011 - 2:16 pm
Is the above still the case (i.e., no test statistic that can compare non-nested models when using wlsmv)? If so, what is the recommended estimator/test statistic combination when comparing non-nested models estimated on the polychoric correlation matrix?
 Linda K. Muthen posted on Monday, April 11, 2011 - 2:42 pm
This is still true as far as I know.
 Catherine posted on Wednesday, January 25, 2012 - 10:52 am
I want to compare two non-nested models of categorical indicators.. but i dont know how.. is it even possible?
 Linda K. Muthen posted on Wednesday, January 25, 2012 - 11:36 am
If you are using maximum likelihood estimation, you can use BIC.
 Patchara Popaitoon posted on Saturday, May 05, 2012 - 12:59 am
Dear Linda,

Please could you advise how to compare non-nested models with all continuous variables. I use MLR estimation for the analysis. Both models in comparison are using the same variables but the structural paths are different. In effect, the only difference in these two models is the number of degree of freedom. Thanks.

pat
 Patchara Popaitoon posted on Saturday, May 05, 2012 - 1:15 am
Dear Linda,

Sorry I gave wrong information in the previous message. Please ignore the previous message.

The structural models in comparision are not nested. They have the same number of manifest variables, but not the same variables. These are all continuous variables. Structural paths are slightly different in both number and direction. Please could you advise how to compare these two models. Thanks.

Pat
 Linda K. Muthen posted on Saturday, May 05, 2012 - 11:54 am
The models cannot be compared. The most you can do is look at fit for each model.
 Patchara Popaitoon posted on Saturday, May 05, 2012 - 12:27 pm
I have come across a webpage explaining that we can use AIC as a measure to identify if one model is better than the other as long as the models in comparison have the same number of manifest variables, but not necessarily the same variables. If this is invalid, what is the boundary condition that we could use AIC for models comparison? Thanks.
 Linda K. Muthen posted on Saturday, May 05, 2012 - 3:02 pm
If you think this is a valid comparison, by all means make it.
 Patchara Popaitoon posted on Saturday, May 05, 2012 - 11:48 pm
Apology for the miscommunication in the previous message. I didn't make it clear. I don't think the guideline provided on that webpage is valid. I feel that a comparison of structural models with different manifest variables is not a valid comparison although these models have the same number of manifest variables (but I might be wrong). I'd like to have your view on this. Also, I'd like to understand when we can use AIC as the measure for models comparison. Do you know any books or papers that would help me undrestand this. Thanks.
 Linda K. Muthen posted on Sunday, May 06, 2012 - 5:20 pm
I believe you need the same set of variables to use BIC or AIC to compare models. This is because without the same variables, they are not in the same metric. See the following FAQ in the website:

BIC citations of interest - how big a difference
 Patchara Popaitoon posted on Sunday, May 06, 2012 - 11:10 pm
Thanks, Linda.
 Naomi White posted on Friday, May 30, 2014 - 6:11 am
Dear Linda,

I wanted to compare the fit of two six-factor CFA models with covariates (using WLSMV estimation). One model has all the covariates loading on six factors while in the second model I've removed all non-significant correlations (including removing one covariate from the model that was not significantly correlated with any factors). Models are below (note, ethn, chgend, sibgend & oldyng are dichotomous). Can I use AIC to compare the fit, or is this not possible because the second model does not contain the Age variable? If it's not possible to compare with AIC for this reason do you know of any references I can cite to explain why I did not compare the models with a test statistic?

Thanks very much
Naomi

MODEL 1:
f1 BY s1 s2 s3;
f2 BY s4 s5 s6;
f3 BY s7 s8 s9;
f4 BY s10 s11 s12;
f5 BY s13 s14 s15;
f6 BY s16 s17 s18;

ethn WITH f1 f2 f3 f4 f5 f6;
chgend WITH f1 f2 f3 f4 f5 f6;
sibgend WITH f1 f2 f3 f4 f5 f6;
oldyng WITH f1 f2 f3 f4 f5 f6;
agegap WITH f1 f2 f3 f4 f5 f6;
age WITH f1 f2 f3 f4 f5 f6;
ses WITH f1 f2 f3 f4 f5 f6;

MODEL 2:
f1 BY s1 s2 s3;
f2 BY s4 s5 s6;
f3 BY s7 s8 s9;
f4 BY s10 s11 s12;
f5 BY s13 s14 s15;
f6 BY s16 s17 s18;

ethn WITH f1 f2 f4 f5 f6;
chgend WITH f1 f3 f4 f5;
sibgend WITH f2;
oldyng WITH f2 f6;
agegap WITH f6;
ses WITH f6;
 Linda K. Muthen posted on Friday, May 30, 2014 - 1:35 pm
Models with the same set of dependent variables can be compared using BIC. But WLSMV does not give BIC. Non-nested models using WLSMV cannot be compared.
 Margarita  posted on Saturday, June 07, 2014 - 10:50 am
Dear Dr. Muthen,

I hope you are well. I understand that BIC/AIC are not provided when WLSMV is used. However, would it be wrong to do it manually for non-nested models?

Thank you
 Linda K. Muthen posted on Saturday, June 07, 2014 - 5:33 pm
Yes, the are based on the loglikelihood which is not available for weighted least squares estimation.
Back to top
Add Your Message Here
Post:
Username: Posting Information:
This is a private posting area. Only registered users and moderators may post messages here.
Password:
Options: Enable HTML code in message
Automatically activate URLs in message
Action: