Fit indices. How is it possible!!! PreviousNext
Mplus Discussion > Categorical Data Modeling >
Message/Author
 Jaume Aguado posted on Thursday, June 03, 2004 - 5:38 am
Hello

I'm comparing the fit of 12 different models fitted via WLSMV and confirmatory factor analysis for ordinal indicators. I have a sample of 380 women and 12 indicators. I'm very surprised at the results I'm getting. All of the models have CFI and TLI greater than 0.95 and SRMR lower than 0.08. But none of them have a RMSEA or WRMR in the recommended limits (<0.06 and <0.90 respectively). At the same time if I compare the unidimensional model with the rest via likelihood ratio test (they are nested), the diference is significant for every model except one.

Is that normal or I'm doing something wrong?

Thanks
 Linda K. Muthen posted on Thursday, June 03, 2004 - 8:08 am
It is sometimes the case that fit measures differ in their assessment of model fit. I wonder how chi-square looks for this model. With a sample size of 380, it should be well-behaving. When you say the likelihood ratio difference tests are significant, I assume that at each step the model is significantly improved?? I don't think that this implies that the final model fits. Version 3 has difference testing for WLSMV.
 Jaume Aguado posted on Thursday, June 03, 2004 - 9:20 am
Yes I mean that every model is an improvement with respect to the unifactorial one. And you are right that doesn't imply that any of the multifactorial models fits the data well. But I'm trying to choose between all the models proposed which of them bests fit my data.
None of the chi-square of goodness of fit (WLSMV) indicates a good fit (p-value>0.05).
I have M-Plus version 2.14 so I can't take advantage of the WLSMV difference testing.
 Linda K. Muthen posted on Thursday, June 03, 2004 - 10:23 am
I would suggest starting with an EFA. Often if you start with a CFA based on theory, and your measures are not valid measures of the constructs intended, then trying to modify a CFA model is fraught with problems. An EFA is a good way to see if you have the number of dimensions intended and if items are behaving as expected.
 Anonymous posted on Friday, August 06, 2004 - 8:18 pm
Is there a reference for the theory behind the new WLSMV difference testing in V3? The technical appendices are not available yet on the link on the home page.

Thanks.
 bmuthen posted on Sunday, August 08, 2004 - 11:59 am
No. We have not yet finished writing up what is programmed here.
 Anonymous posted on Sunday, August 08, 2004 - 3:15 pm
Thanks for the quick reply re the difference testing. in the interim, how should this method be referenced?
 bmuthen posted on Sunday, August 08, 2004 - 3:19 pm
It is probably best to simply refer to the Version 3 User's Guide.
 Mahyar posted on Wednesday, November 03, 2004 - 2:56 pm
Hello,

I have ran a CFA of categorical (ordinal) data with WLSMV estimator. The underlying factor has six indicators. Data are non-normally distributed and N is 628. Fit indices are:

Chi square: 98.369, P vale is significant
CFI: .974
TLI: .989
RMSEA: .144
SRMR: .030
WRMR: 1.683

I have several questions:

(1) Which fit indices should I report?
(2) Why is RMSEA high? Should I even report it? I heard that RMSEA may NOT be a good fit index to use for categorical data.
(3) What is a good/acceptable range for WRMR? And finally,
(4) Why for a CFA the output gives SRMR result, but not for a full SEM model?

Thank you.

Mahyar
 Linda K. Muthen posted on Wednesday, November 03, 2004 - 3:47 pm
There is a dissertation on our website by Yu. See Mplus papers. Here she discusses fit measure cutoffs for models with categorical outcomes. I suggest that you read her recommendations. SRMR does not come out when there are independent variables in the model.
 Martin Brunner posted on Monday, November 08, 2004 - 2:09 am
Hello,

could you help me with a chi2-difference question, please .
I ran a CFA (N=357) with four correlated factors (M1). Each factor was measured by 10 indicators and each indicator loaded on only one factor. Estimation method was WLSM.
I was interested wether one particular factor intercorrelation was significantly different from 1.0. Thus I ran a separate CFA imposing that restriction (M0).
I tried to calculate the chi2-difference according to the formula provided in the technical appendix (p.22, formula 120 and 121). Following are the results:

chi2/df/Scalingfactor/chi2-Difference
M1: 1108.0/734/0.749
M0: 1182.2/735/0.748/-3880.7

I wonder, wether the chi2-difference is not a bit too large.

Your help is highly appreciated.

Best wishes

Martin
 bmuthen posted on Sunday, November 14, 2004 - 12:56 pm
Do you get the same result using the Version 3 "DIFFTEST" option?

If that also gives unexpected results, please send your input, output and data to support@statmodel.com.

I am not sure that a nested model that has a parameter value on the border of the admissible parameter space, such as a unit factor correlation, would allow good chi-square difference testing.
 Anonymous posted on Thursday, November 25, 2004 - 2:11 am
Dear Bength,

thanks a lot for your advice.

When I used the DIFFTEST with WLSMV the chi2 looked pretty "normal" compared to the WLSM results. The estimated correlation for the the unrestricted model M1 was .77 (chi2 = 288.331, df=191). By fixing this correlation to 1.0 (Model M0) the resulting chi2 was 307.199 (df=191). The chi2-difference as estimated by DIFFTEST was 36.737 (df=1).

Concerning the appropriateness of test. I believed that testing, wether a two factor model (M1) is a better approximation compared to a one-factor model (M0) implies to fix the factor correlation to 1.0 in M0. But I am definitely no expert on chi2-difference testing.

Best wishes

Martin
 bmuthen posted on Friday, November 26, 2004 - 6:42 am
Yes, fixing a factor correlation to 1 is a common approach to this. But that may cause the chi-square approximation to not be good. There is a general rule that 2 nested models do not give a proper chi-square distribution if the nested model has a parameter value on the border of the admissible parameter space, such as a unit factor correlation. How approximate it is in your case, I don't know (it calls for a monte carlo simulation study). If you want to report these results, you should mention this potential problem, and perhaps show the results as descriptive evidence not inferential evidence.
 Anonymous posted on Thursday, May 05, 2005 - 1:43 am
I'm interested in this "value on the border" issue. Is there any reference about it? And, is there a reference of a montecarlo simulation of that same issue? I mean how good is the chi-square difference test when a restriction has a value on the border...

Thank you
 Linda K. Muthen posted on Thursday, May 05, 2005 - 6:06 am
See the following reference for a discussion of parameters on the border:

Lo, Mendel, & Rubin (2001). Testing the number of components in a normal mixture. Biometrika, 88, 767-778.
 Lange Gasse 20; 90403 nuremberg, Ge posted on Friday, January 26, 2007 - 12:14 pm
Dear Madam or Sir,
I have a question concerning fit indices for CFA with categorical variables and 4 groups using WLS (20.000 subjects).
When I set factor loadings and thresholds free in each group, CFI increases as expected, but RMSEA increases and TLI decreases. Should I ignore the latter two or does this indicate something else?

sample in/output follows.
thank you very much
Hage Wolff

****INPUT full invariance
Data: ngroups = 4; ...
Variable: names are i2 t1 t3 i4 t5 o4 o5 i6 o8;
Categorical = all;
Analysis: Type = mgroup meanstructure;
Estimator = WLS;
para =delta;
Model: F1 by i2 i4 i6 t1 t3 t5 o4 o5 o8;
****OUTPUT
Chi-Square Model Fit
Value 5218.899/ df 210
Baseline Model
Value 89035.852/ df 144
CFI 0.944
TLI 0.961
RMSEA 0.070

**** INPUT free factor loadings & thresholds
Model: F1 by i2 i4 i6 t1 t3 t5 o4 o5 o8;
Model group2: F1 by i2 i4 i6 t1 t3 t5 o4 o5 o8;
F1 by i2@1;
[i2$3 i2$4];
[i4$2 i4$3 i4$4];
[i6$2 i6$3 i6$4];
[t1$2 t1$3 t1$4];
[t3$2 t3$3 t3$4];
[t5$2 t5$3 t5$4];
[o4$2 o4$3 o4$4];
[o5$2 o5$3 o5$4];
[o8$2 o8$3 o8$4];
[similar for groups 3 & 4]

Chi-Square Model Fit
Value 3720.447/ df 111
Baseline Model
Value 89035.852 / df 144
CFI 0.959
TLI 0.947
RMSEA 0.082
 Bengt O. Muthen posted on Friday, January 26, 2007 - 6:15 pm
The fit indices do not always change in tandem. This is clear in simulation studies such as Yu's 2002 dissertation which is on our web site under Papers. She found that CFI is often more dependable than other measures.
Back to top
Add Your Message Here
Post:
Username: Posting Information:
This is a private posting area. Only registered users and moderators may post messages here.
Password:
Options: Enable HTML code in message
Automatically activate URLs in message
Action: