Anonymous posted on Wednesday, January 28, 2004 - 6:36 am
When running a CFA on 8 vars I have i) model 1: a 1 factor model - all 8 vars load onto a single factor. ii) model 2: a 2 factor model - 4 vars on 1 fac, 4 vars on the other. iii) model 3: a 3 factor orthogonal model; 3 vars on 1 factor, 3 on another, 2 on a third AND all corrs between factors set = 0. iv) model 4: 3 factor oblique model (as model 3 but corrs allowed between factors). v) model 5: a 2nd order 3 factor model - as iii) but the three factors then load onto a single factor.
Am I right in saying that model iii) is nested within model iv), but there is no other nesting here?
With 7 observed variables, I have 2 models: 1) all 7 items load on 1 factor 2) 4 items load on factor A and 3 items on factor B. Factors A and B are not correlated (fixed at 0). Is it correct that the models are not nested (they have the same # of df's) and chi-square difference test is not allowed. If this is correct, can I still compare the other fit statistics (GFI etc)
I don't believe these models are nested. I don't know of any ways to compare non-nested models using various fit statistics other than descriptively.
Kerry Lee posted on Tuesday, November 30, 2010 - 12:41 am
Dear Dr Muthen,
I have been going through the posts on the testing of nested CFA models and am getting a bit confused.
The scenario that I am concerned with involves testing whether a number of observed variables fit a 1, 2, or 3 factor CFA model (similar to the example given in .../9/1611.html). In a reply to that post, it was stated that forcing the correlation between two factors to 1 violates a key assumption in testing nested model using chi-squared.
In ...9/344.html, similar advice was given: a 2 factor model in such scenarios is not nested under a 3 factor model.
Yet, in post .../9/5167.html, an elegant solution seems to be available using the MODEL TEST command. Rather than fix the correlation between two factors to 1, a Wald test can be used to test whether the the correlation between two factors differ significantly from 1.
I guess if the Wald test is not significant, one can then infer that the correlation in question does not differ from zero and adopt the model with fewer factors.
Because the Wald test is applied to only one model, the question of "nestedness" does not technically arise. Can I please confirm whether this line of reasoning is correct and that using the Wald test in this way allows for the evaluation of a 2 versus 3 factor model?
Testing on the border of the admissible parameter space (corr=1 being one example, v=0 another) could affect both LRT chi-square testing and Wald testing (z testing in a univ setting). But the question remains how well or poorly it works in practice. Next, I give input for 2 simulation studies I did to explore testing 1 factor vs 2 CFA factors. You can run it and look at the good results for the Wald test. And then generalize to your setting.
Here is Monte Carlo input for generating data from a 1-factor model and analyzing with a 2-factor CFA. The question is what the empirical p-value is for the Wald test: How often do we reject that the 2 factors correlate one? The p-value is 0.06 which is close to the correct 0.05, that is, we reject that the factor correlation is one at the correct low Type I error level. In other words, we correctly choose 1 factor. The p-value 0.06 is also seen in the last power column for the New parameter diff.
title: 1f 2f montecarlo: names = y1-y8; nobs = 200; nreps = 500; model population: f by y1-y8*.8; f*1; y1-y8*.36; model: f1 by y1-y4*.8; f2 by y5-y8*.8; f1-f2@1; f1 with f2*1 (corr); y1-y8*.36; model constraint: new(diff*0); diff = 1-corr; model test: 0 = 1-corr;
Here is Monte Carlo input for generating data from a 2-factor CFA and analyzing with the same model. Here the empirical 5% p-value and power estimate is 0.74 at n=200, which approaches good power to reject the false hypothesis that the factors correlate 1. Note that this is the case despite the true correlation being high, 0.95.
title: 2f 2f montecarlo: names = y1-y8; nobs = 200; nreps = 500; model population: f1 by y1-y4*.8; f2 by y5-y8*.8; f1-f2*1; f1 with f2*.95; y1-y8*.36; model: f1 by y1-y4*.8; f2 by y5-y8*.8; f1-f2@1; f1 with f2*.95 (corr); y1-y8*.36; model constraint: new(diff*.05); diff = 1-corr; model test: 0 = 1-corr;
Kerry Lee posted on Wednesday, December 01, 2010 - 7:41 pm
Dear Dr. Muthen,
Thank you for the detailed response. I will work through the examples as soon as I finish reading the Monte Carlo section in the Users' Guide!
I have a related question. In my data set, one of the variables is censored. For this reason, I have been using the MLR estimator and using the S-B scaled chi-squared. Unfortunately, the data is such that I have to use the strictly positive version. Again, the focus of the analysis is whether a 1, 2, or 3 factor model better explains the data. I am running this analysis on both longitudinal and cross-sectional samples, with a view to linking up the samples in an accelerated design/growth model later on.
Is there any precaution that I should take in generalizing Monte Carlo findings regarding the Wald test to the S-B strictly positive chi-squared?
MLR is robust to non-normality and this carries over to the Wald test.
I don't know how much you can generalize these results. You would need to do a study for each situation to know for sure.
Kerry Lee posted on Friday, December 03, 2010 - 2:59 am
Dear Dr. Muthen,
Thanks for the reply. I hope you do not mind clarifying one more thing. I gather from posts elsewhere that the Wald test can legitimately be used with MLR to test the effect of constraining a parameter (rather than the SB scaled chi squared).
I also read that the Wald and the chi-squared difference tests are supposed to be asymptotically equivalent. If this is the case, how come the Wald is not affected by the same issues that render the chi-squared unsuitable for analyses conducted with a MLR estimator?
It is true that these two methods are asymptotically equivalent. Methods that are asymptotically equivalent may have different sensitivities. If you did a Monte Carlo study of likelihood ratio chi-square difference testing for your situation, you can see if in your case this test is also not sensitive. However, both tests are theoretically not correct for testing parameters on the border of the admissible parameter space.
Dear, I've been trying to understand an article by Schönberger in Psychiatry Research. The authors test the Hospital Anxiety & Depression Scale (HADS) to see whether the 1-, 2- or 3-factor structure fits the data best. Model 1 = 1 factor (14 items) Model 2 = 2 factors (depression = 7 items & anxiety = 7 items ) Model 3 = 3 factors (depress = 7 items, anxty = 3 items, neg affectivity = 4 items) + 2 arrows departing from affectivity factor to the anxiety and depression factors.
Now the authors state that Model 1 is nested in Model2 & Model3 + Model2 is nested in the 3 factor model. This to do X² diff testing.
However, if I want to do the same thing (difftest because I use WLSMV), MPlus indicates that the 1factor model is not nested within the 2factor model. Can you please clear this up because I do not understand this? A 2nd problem are the DFs. According to my outputs, model 1 has a X² of 111.5 and DF of 76, model 2 has a X² of 111.6 and DF of 75 and model 3 has a X² of 97.3 and DF of 73. The DF thus go down while the paper of Schonberger reports the DF to go up from 1 to 3 factors).
If you would need the paper, the reference is:
Schönberger & Ponsford(2009), 'The factor structure of the Hospital Anxiety and Depression Scale in individuals with traumatic brain injury' in Psychiatry Research, 179: 342-349
Dear Dr Muthen, For a set of 14 variables, I want to test whether there is one underlying factor. Whereas a simple one-factor model did not fit well, I successively added 4 (theoretically plausible) residual correlations to the model. I interpret the resulting model A as indicating one-dimensionality with minor additional factors. I am wondering whether in order to better support this, I should compare model A to a model B specifying 5 correlated factors, the hypothesis being that in a chi2-difference test model A fit is not significantly worse than that of the more liberal model B.
Model A would look like this: ERA by item1 item2 item3 item4 item5 item6 item7 item8 item9 item10 item11 item12 item13 item14; item1 with item2; item3 with item4; item5 with item6; item7 with item8;
Model B: ERA by item9 item10 item11 item12 item13 item14; FAC1 by item1 item2; FAC2 by item3 item4; FAC3 by item5 item6; FAC4 by item7 item8;
My question relates to the Monte Carlo code Dr. Muthen posted here on 12/1/10. I'm trying to adapt that code to do a power analysis for a multi-trait, multi-method CFA in which I have two correlated traits and two correlated methods. Specially, I want to test whether the correlation between the two traits and that between the two methods are significantly different from 1.
The problem is that even with 10,000 obs, the power estimates for my code are in the range of .69-.71 for diff1 and diff2, which seems impossible.
I'm running out of space so will post my code in a new post.
Any help you can offer would be greatly appreciated.
model population: fMethod1 by y1-y4*.8; fMethod2 by y5-y8*.8; fMethod1*1; fMethod2*1; fTrait1 by y1*.8 y3*.8 y5*.8 y7*.8; fTrait2 by y2*.8 y4*.8 y6*.8 y8*.8; fTrait1*1; fTrait2*1; y1-y8*.36; fMethod1 with fMethod2*.80; fTrait1 with fTrait2*.80;
analysis: model = nocovariances;
model: fMethod1 by y1-y4*.8; fMethod2 by y5-y8*.8; fMethod@1fMethod2@1; fTrait1 by y1*.8 y3*.8 y5*.8 y7*.8; fTrait2 by y2*.8 y4*.8 y6*.8 y8*.8; fTrait1@1fTrait2@1; y1-y8*.36; fMethod1 with fMethod2*.80 (corr1); fTrait1 with fTrait2*.80 (corr2);
model constraint: new(diff1*.20); new(diff2*.2); diff1 = 1-corr1; diff2 = 1-corr2;
model test: 0 = 1-corr1; 0 = 1-corr2;
Emil Coman posted on Wednesday, August 24, 2011 - 6:23 am
Dear Michael, I took the liberty of running your syntax [thanks for posting, I saved and will use it in future analyses...]. After correcting a small typo on line fMethod@1fMethod2@1; which is fMethod1@1fMethod2@1; it ran well, and got, I think as expected, for the Wald Test of Parameter Constraints Proportions Percentiles Expected Observed Expected Observed 0.050 1.000 5.991 2357.723 If we tweak the instructions on p 362 in the guide, 'the column 1 value of 0.05 gives the probability that the Wald Test value exceeds the column 3 percentile value of 5.991. Columns 2 and 4 give the corresponding values observed in the Monte Carlo replications. Column 2 gives the proportion of replications for which the critical value is exceeded,' which here is 100%. So the Wald test concludes that both constraints 1-corr1=0 and 1-corr2=0 are rejected, powerfully so, right? The new parameters indicate that .8's are different than 1, and .2's are not different than 0. Is this what you expected? Am I right on this? Cheers, Emil
Hi Emil, Thanks for your help. I apologize for the typo; it wasn't in the code I was running but got introduced when I was renaming the factors to make them more intelligible here.
Anyway, my confusion is because the power estimate from the Wald Test of Parameter Constraints that you mention is so widely different from the power estimates given by the % Sig coeff column under MODEL RESULTS for the individual parameters diff and diff2, which are .697 for diff and .704 for diff2 when n = 10,000. I wouldn't expect that these would be identical since the Wald test is a global test of two constraints, but am surprised they are so different. Also, I'm interested in testing these individually.
To make matters more confusing, the power given by the Wald test of Parameter Constraints for this model seems strikingly high even at small sample sizes (e.g., .993 when n = 100), so I'm wondering if there's something wrong with the factor specification. Or does this sound reasonable to other folks?
I ran your setup with n=10000 and 500 replications. The output shows that only of 442 of the 500 reps converged without problems and Tech9 shows the various estimation problems in many replications. The coverage is poor and there is both parameter and SE estimate bias. The power estimates in the last column (%Sig) rely on these quantities being well estimated. So the results are not trustworthy. Not for Wald either. The model is probably not a stable one, which we know is the case for MTMM with few methods.
1. You don't need to hold the loadings of the 2 indicators equal for identification in Model 1. You would need to do that if the factors are specific factors in a model with a general factor, as in Model 3. Note also that this is not how to hold them equal - you should say
v by v1-v2* (lv1);
2. Model 2 is equivalent with model 1 because you have only 3 factors as indicators.
3. I believe Model 2 is nested within Model 3 when you have enough indicators. For a discussion, see e.g.
Yung, Thissen & McLeod (1999). On the relationship between higher-order factor model and the hierarchical factor model. Psychometrika, 64, 113-128.
I“ve got a questionnaire and two different CFAs (three factors each, but the factos consist of different variables) One CFA consists of 25 Items, the other one only contains of 21 of these 25 Items. I treated the items as ordered categorical. The data are not normally distributed. I used the WLSMV-estimtor. Am I right that these two models are not nested? (So I can“t use chi-square-difference test). What can I do to test the difference between these two models?
Ok, I didn“t know about that! What do you think about reestimating both CFAs in a new sample using the WLSM-extimator or Maximum-likelihood. Then comparing them with AIC or BIC would that be possible (if I still wouldn“t use exactly the same set of obeserved variables)? Would that make sense at all?
All the WLS estimators are weighted least squares. When you have a different set of observed variables, AIC and BIC are not in the same metric.
Paula Vagos posted on Friday, December 07, 2012 - 7:59 am
Dear Dr. Muthen I am qoeking with a data set of 549 observations on 40 variables, and I wanted to try different CFA on possible organization of those variables. I tried 3 models: 1. 40 variables loading on factor A 2. 20 variables loading on factor A and 20 on factor B 3. 10 variables loagind on factor A, 10 on factor B, 10 on factor c, and 10 on factor D.
I though theses were nested models, and so I cousl use the chi-square difference test (the scaled one, because my data is non-normal) to compare de models. Is this correct?
Jane Wu posted on Wednesday, December 12, 2012 - 10:36 pm
OK, now I see that Models A and C have the same df and can't be compared.
Model A, however, does have 2 more df than Model B. As I understand it, models with all first-order factors subsumed by one second-order factor (e.g., Model B) are nested within models where all first-order factors are intercorrelated (Model C). Using that rationale, I was wondering if the nesting might work like this:
- 3 of the 4 factors could be subsumed by a higher-order factor (in Model A) - And the correlations between the 4th factor and 2 remaining factors are fixed at zero - Note that I'm still estimating one correlation between the 4th factor and a 1st/2nd order factor
Maybe this is a stretch? In which case, I guess comparing AIC and BIC is the way to go. Thanks, Bengt!
Note that in Model A you have a just-identified part where you have 3 1st-order factors influenced by 1 2nd-order factor.
The way to find out if a model X is nested within a model Y is to see if you can restrict parameters of model Y to get model X (get it to fit the same as model X. So you can experiment along those lines. See also a 2010 Bentler-Satorra article in Psych Methods on the topic of how to decide nestedness.
JOEL WONG posted on Friday, August 09, 2013 - 7:30 am
I want to compare 2 measurement models -- Model A is a bifactor model with a general factor and 2 group factors (general factor is orthogonal to the group factors and the group factors are orthogonal to each other). Model B only has two correlated group factors. Is Model B nested within Model A? I'm guessing not, but just want to be sure. I used MLMV and when I ran the chi square dff test, there was a message indicating that model B may not be nested in A. I read a study that used the chi square diff test to compare these 2 models, implying that they are nested.
If you use EFA for the bi-factor and 2-factor models, they are nested. This is because the EFA bi-factor model with 2 specific factors (2 group factors) has the same fit as a regular 3-factor EFA and a regular 2-factor EFA is nested within a regular 3-factor EFA.
For the CFA versions, I am not sure and would have to see the two outputs. A not nested message is given if the model with more parameters does not have a better loglikelihood (or WLSMV fitting function). Getting stuck in a local solution can be one explanation.
JOEL WONG posted on Saturday, August 10, 2013 - 1:16 am
Bengt, thanks a lot. I used CFA. As you rightly predicted, the model with more parameters (bifactor model) has a poorer loglikelihood. I'm guessing this is because in my bifactor model, the 2 group factors are specified as orthogonal whereas in my 2-factor model, the 2 group factors are allowed to correlate with each other. So,the 2-factor model is NOT tested within the bifactor model, and I should be looking at the BIC values rather than the chi square difference to assess model fit. Am I right?
I am comparing each of three alternative measurement models (unidimensional, correlated factors, and second-order model) to a fourth (the bifactor model). The unidimensional is clearly nested with the bifactor model. The second-order model is also nested within the bifactor model as well per Yung et al., 1999.
Riese (2012, MBR) has noted that the correlated factors model is nested within the bifactor model; as a consequence, he uses a chi-square difference test to compare these models. My concern is that when you go from the bifactor model to the correlated traits model, you have to constrain all general factor loadings to zero AND then allow the first-order factors to correlate. It is this latter part of removing the orthogonality restrictions on the specific factors that makes me question the utility of the chi-square difference test in this case (i.e., the models may not be nested). Mplus will return the DIFFTEST in my case but I am unsure of whether to put any stock in it. Any guidance would be much appreciated. Thanks.
Recall that a bi-factor EFA with m-1 specic factors is equivalent to EFA with m factors, that is, the two types of models have the same fit to data. The two models are reparameterizations of each other, so that the bi-factor EFA represents a special type of rotation. With that as background, I would not share your concern about removing the orthogonality restrictions. I have not studied this, but I would not a priori think that nesting doesn't hold. Analogous situations arise with mean structure modeling - new parameters may be added to the more restrictive model, but the model may still be nested.
Eiko Fried posted on Thursday, July 03, 2014 - 7:07 am
It has been my understanding so far that factor solutions with different number of factors are not nested and cannot be compared using a chi-square or similar tests. In this case, absolute and relative fit indices have to be taken into account (BIC, RMSEA, etc).
I recently found this http://davidakenny.net/cm/mfactor.htm: "A one-factor model is nested within a two-factor as a one-factor model can be viewed as a two-factor model in which the correlation between factors is perfect). "
Would that be a feasible approach to compare 1 and 2 factor models with a chisquare test in Mplus by estimating the 1 factor model as 2 factor model and correlating it to 1?
I'm getting a little confused about two models. Initially I thought they were nested and MPLUS will calculate DIFFTEST for them. However, on closer inspection I am unsure if they truley are nested. The models are:
MODEl: F1 BY y2 y5 y7 y8 y9 y10 y11 y13 y1; F2 BY y3 y6 y12 y4;
MODEl2: F1 BY y5 y7 y11 y1 y9; F2 BY y4 y6 y10 y13 y2 y8; F3 BY y12 y3;
is DIFFTEST incorect in treating these as nested or have I made an error here? (p.s. these are ordinal indicators, although I don't think that makes a difference here). Thanks
My understanding from reading this forum is that to make the models "nested" I have to start with the most complex (M3: 3 factor model), and then for each simpler model, I have to fix the correlation between two factors at 1. So my models would look like this:
Model 3: PS BY HF2b FL2b MSIT2b; IC BY HF2 FL2 FL4 MSIT2; CF BY HF5 FL7; PS WITH IC CF; IC WITH CF;
Model 2 PS BY HF2b FL2b MSIT2b; IC BY HF2 FL2 FL4 MSIT2; CF BY HF5 FL7; PS WITH IC CF; IC WITH CF@1.0;
Model 1 PS BY HF2b FL2b MSIT2b; IC BY HF2 FL2 FL4 MSIT2; CF BY HF5 FL7; PS WITH IC@1; PS WITH CF@1; IC WITH CF@1;
When I run this code, the model does not converge. I get an error saying: NO CONVERGENCE. SERIOUS PROBLEMS IN ITERATIONS. ESTIMATED COVARIANCE MATRIX NON-INVERTIBLE. CHECK YOUR STARTING VALUES.
Is my conceptual understanding of nesting accurate (i.e. fixing the correlation between factors to 1)? Is my code to fix the correlations between factors accurate? If so, why is my model not converging?
Thank you so much for any guidance you can provide.
If you are comparing the number of factors I would instead use BIC.
Jiyeon So posted on Tuesday, June 21, 2016 - 9:04 pm
Hi Dr. Muthen,
I tested four structural models in the process of validating a scale:
Model1 (1st order four factor): A B C D (only covariances among them are estimated) Model 2 (2nd order one factor): F by A B C D; Model 3 (2nd order two factor): X by A B; Y by C D; X with Y; Model 4 (3rd order one factor): F by X Y; X by A B; Y by C D;
I am particularly interested in whether model 4 is in nested relationships with another model (so that I determine whether or not I can use chisquare difference test to compare the model fit).
Dear Dr. Muthen, I am running four different CFAs with the same data set using WLSMV. The variables are categorical. The models are: 1-factor, correlated 4 factors, second order with 4 correlated factors and a bifactor. My aim is to compare the bifactor model with the other three seperately. And so;
1) Are the other three models nested in bifactor? 2) If there is one (or more) non-nested model, how will i compare its chi-square value with bifactor and decide which one fits the data best?