Comparing non-nested models PreviousNext
Mplus Discussion > Categorical Data Modeling >
Message/Author
 Jim Prisciandaro posted on Tuesday, April 12, 2005 - 1:54 pm
Hello,

I wish to compare 2, 3, and 4 factor CFA models which are (I believe) nonnested. These are all "standard" CFA models and all share the same exact variable pool (and all variables are ordered binary). However, the models which contain more factors are not created by simply splitting the factors from the models which contain fewer factors. For example, in the two factor model the following is specified:

MODEL: f1 BY lintrst decapp decwate incapp incwate earlins midins latins hypins fatigue divar retard agit pleasur lsex conc thgt indeci;
f2 BY worthls sinful guilty conf inferio death wdie commit attempt;

In the three factor model:

MODEL: f1 BY decapp decwate earlins midins latins agit;
f2 BY death wdie commit attempt;
f3 BY lintrst hypins fatigue divar retard pleasur lsex conc thgt indeci incapp incwate worthls sinful guilty conf inferio;

Finally, in the four factor model:
MODEL: f1 BY decapp decwate earlins midins latins divar retard agit;
f2 BY death wdie commit attempt;
f3 BY incapp incwate;
f4 BY lintrst hypins fatigue pleasur lsex conc thgt indeci worthls sinful guilty conf inferio;

So, my first question is: Are (any of) these models nested? If so, how?

Second, if they are not nested (which is my understanding), is there any way I can compare the fit of these models with one another? I tried to use the MLR estimator in order to obtain BIC values for each of these models, but the 4-factor model would not estimate (error was something to the effect of: not enough physical memory, over 50,000 integration points required), and the 3-factor model probably would have estimated but it would have took days to weeks for the computer to estimate the model.

Can I calculate the BIC (or AIC) by hand with the WLSMV estimator? If so, how? If not, how would you compare the fit of these models?

Thank you so much,
Jim
 BMuthen posted on Thursday, April 14, 2005 - 12:04 am
I don't believe that these models are nested. You can use BIC. To make the three and four factor models computationally feasible, reduce the number of integration points by saying INTEGRATION = 5; to get an approximate ML solution. You cannot get BIC via WLSMV because you need the loglikelihood value which you can only get through maximum likelihood. Be sure you are using Version 3.12 as there have been significant speed improvements for multiple integration.
 Jason Fogler posted on Friday, June 02, 2006 - 2:19 pm
I was so glad to come across this thread as I am dealing with a very similar situation (comparing nonnested 2- and 3-factor categorical CFA models). Dr. Muthen, could you please explain how reducing the number of integration points permits the use of ML and BIC?

Many Thanks -- Jason
 Linda K. Muthen posted on Friday, June 02, 2006 - 3:06 pm
With maximum likelihood and categorical outcomes, numerical integration is required. You can read about numerical integration on pages 330-333 of the Mplus User's Guide. Reducing the number of intergration points makes the analysis computationally feasible when there are more than a couple of factors.
 Sarah Olivo posted on Wednesday, April 15, 2009 - 8:43 am
I am attempting to compare non-nested, second-order models which include continuous and categorical data. Model specification is listed below. Based on my understanding of your statement above, "Reducing the number of intergration points makes the analysis computationally feasible when there are more than a couple of factors," am I to assume that using numerical integration is not feasible for my purposes since I have either 1 or 2 higher order factors?

Also, theoretically it seems that these models should be nested, although using the DIFFTEST function, Mplus tells me they're not.

Syntax:
MODEL: MDD BY pmdd3 pmdd4 pmdd5;
GAD BY pargad1 pargad2 pargad3;
SOC BY parsoc1 parsoc2 parsoc3;
PANIC BY parpd2 parpd6 parpd7;
OCD BY parob2 parob3 parob9;
NA BY MDD GAD SOC PANIC OCD;

MODEL: MDD BY pmdd3 pmdd4 pmdd5;
GAD BY pargad1 pargad2 pargad3;
SOC BY parsoc1 parsoc2 parsoc3;
PANIC BY parpd2 parpd6 parpd7;
OCD BY parob2 parob3 parob9;
NA BY MDD GAD SOC PANIC OCD;
MDD WITH SOC;
 Linda K. Muthen posted on Friday, April 17, 2009 - 10:43 am
Please send the two outputs and your license number to support@statmodel.com.
 Michael S. Businelle, Ph.D. posted on Monday, February 15, 2010 - 2:16 pm
Hello.

I am trying to determine which of several non-nested models is the best fit for my data. The outcome variable is binary (yes/no) so I used the WLSMV estimator. The same variables are being used for all models.

How does one go about determining the best fitting model in this situation?
 Linda K. Muthen posted on Monday, February 15, 2010 - 3:39 pm
With weighted least squares, I know of no test statistic that can compare non-nested models.
 Jamie Marincic posted on Monday, April 11, 2011 - 2:16 pm
Is the above still the case (i.e., no test statistic that can compare non-nested models when using wlsmv)? If so, what is the recommended estimator/test statistic combination when comparing non-nested models estimated on the polychoric correlation matrix?
 Linda K. Muthen posted on Monday, April 11, 2011 - 2:42 pm
This is still true as far as I know.
 Catherine posted on Wednesday, January 25, 2012 - 10:52 am
I want to compare two non-nested models of categorical indicators.. but i dont know how.. is it even possible?
 Linda K. Muthen posted on Wednesday, January 25, 2012 - 11:36 am
If you are using maximum likelihood estimation, you can use BIC.
 Patchara Popaitoon posted on Saturday, May 05, 2012 - 12:59 am
Dear Linda,

Please could you advise how to compare non-nested models with all continuous variables. I use MLR estimation for the analysis. Both models in comparison are using the same variables but the structural paths are different. In effect, the only difference in these two models is the number of degree of freedom. Thanks.

pat
 Patchara Popaitoon posted on Saturday, May 05, 2012 - 1:15 am
Dear Linda,

Sorry I gave wrong information in the previous message. Please ignore the previous message.

The structural models in comparision are not nested. They have the same number of manifest variables, but not the same variables. These are all continuous variables. Structural paths are slightly different in both number and direction. Please could you advise how to compare these two models. Thanks.

Pat
 Linda K. Muthen posted on Saturday, May 05, 2012 - 11:54 am
The models cannot be compared. The most you can do is look at fit for each model.
 Patchara Popaitoon posted on Saturday, May 05, 2012 - 12:27 pm
I have come across a webpage explaining that we can use AIC as a measure to identify if one model is better than the other as long as the models in comparison have the same number of manifest variables, but not necessarily the same variables. If this is invalid, what is the boundary condition that we could use AIC for models comparison? Thanks.
 Linda K. Muthen posted on Saturday, May 05, 2012 - 3:02 pm
If you think this is a valid comparison, by all means make it.
 Patchara Popaitoon posted on Saturday, May 05, 2012 - 11:48 pm
Apology for the miscommunication in the previous message. I didn't make it clear. I don't think the guideline provided on that webpage is valid. I feel that a comparison of structural models with different manifest variables is not a valid comparison although these models have the same number of manifest variables (but I might be wrong). I'd like to have your view on this. Also, I'd like to understand when we can use AIC as the measure for models comparison. Do you know any books or papers that would help me undrestand this. Thanks.
 Linda K. Muthen posted on Sunday, May 06, 2012 - 5:20 pm
I believe you need the same set of variables to use BIC or AIC to compare models. This is because without the same variables, they are not in the same metric. See the following FAQ in the website:

BIC citations of interest - how big a difference
 Patchara Popaitoon posted on Sunday, May 06, 2012 - 11:10 pm
Thanks, Linda.
Back to top
Add Your Message Here
Post:
Username: Posting Information:
This is a private posting area. Only registered users and moderators may post messages here.
Password:
Options: Enable HTML code in message
Automatically activate URLs in message
Action: