I'm comparing the fit of 12 different models fitted via WLSMV and confirmatory factor analysis for ordinal indicators. I have a sample of 380 women and 12 indicators. I'm very surprised at the results I'm getting. All of the models have CFI and TLI greater than 0.95 and SRMR lower than 0.08. But none of them have a RMSEA or WRMR in the recommended limits (<0.06 and <0.90 respectively). At the same time if I compare the unidimensional model with the rest via likelihood ratio test (they are nested), the diference is significant for every model except one.
It is sometimes the case that fit measures differ in their assessment of model fit. I wonder how chi-square looks for this model. With a sample size of 380, it should be well-behaving. When you say the likelihood ratio difference tests are significant, I assume that at each step the model is significantly improved?? I don't think that this implies that the final model fits. Version 3 has difference testing for WLSMV.
Yes I mean that every model is an improvement with respect to the unifactorial one. And you are right that doesn't imply that any of the multifactorial models fits the data well. But I'm trying to choose between all the models proposed which of them bests fit my data. None of the chi-square of goodness of fit (WLSMV) indicates a good fit (p-value>0.05). I have M-Plus version 2.14 so I can't take advantage of the WLSMV difference testing.
I would suggest starting with an EFA. Often if you start with a CFA based on theory, and your measures are not valid measures of the constructs intended, then trying to modify a CFA model is fraught with problems. An EFA is a good way to see if you have the number of dimensions intended and if items are behaving as expected.
Anonymous posted on Saturday, August 07, 2004 - 2:18 am
Is there a reference for the theory behind the new WLSMV difference testing in V3? The technical appendices are not available yet on the link on the home page.
bmuthen posted on Sunday, August 08, 2004 - 5:59 pm
No. We have not yet finished writing up what is programmed here.
Anonymous posted on Sunday, August 08, 2004 - 9:15 pm
Thanks for the quick reply re the difference testing. in the interim, how should this method be referenced?
bmuthen posted on Sunday, August 08, 2004 - 9:19 pm
It is probably best to simply refer to the Version 3 User's Guide.
Mahyar posted on Wednesday, November 03, 2004 - 8:56 pm
I have ran a CFA of categorical (ordinal) data with WLSMV estimator. The underlying factor has six indicators. Data are non-normally distributed and N is 628. Fit indices are:
Chi square: 98.369, P vale is significant CFI: .974 TLI: .989 RMSEA: .144 SRMR: .030 WRMR: 1.683
I have several questions:
(1) Which fit indices should I report? (2) Why is RMSEA high? Should I even report it? I heard that RMSEA may NOT be a good fit index to use for categorical data. (3) What is a good/acceptable range for WRMR? And finally, (4) Why for a CFA the output gives SRMR result, but not for a full SEM model?
There is a dissertation on our website by Yu. See Mplus papers. Here she discusses fit measure cutoffs for models with categorical outcomes. I suggest that you read her recommendations. SRMR does not come out when there are independent variables in the model.
could you help me with a chi2-difference question, please . I ran a CFA (N=357) with four correlated factors (M1). Each factor was measured by 10 indicators and each indicator loaded on only one factor. Estimation method was WLSM. I was interested wether one particular factor intercorrelation was significantly different from 1.0. Thus I ran a separate CFA imposing that restriction (M0). I tried to calculate the chi2-difference according to the formula provided in the technical appendix (p.22, formula 120 and 121). Following are the results:
I am not sure that a nested model that has a parameter value on the border of the admissible parameter space, such as a unit factor correlation, would allow good chi-square difference testing.
Anonymous posted on Thursday, November 25, 2004 - 8:11 am
thanks a lot for your advice.
When I used the DIFFTEST with WLSMV the chi2 looked pretty "normal" compared to the WLSM results. The estimated correlation for the the unrestricted model M1 was .77 (chi2 = 288.331, df=191). By fixing this correlation to 1.0 (Model M0) the resulting chi2 was 307.199 (df=191). The chi2-difference as estimated by DIFFTEST was 36.737 (df=1).
Concerning the appropriateness of test. I believed that testing, wether a two factor model (M1) is a better approximation compared to a one-factor model (M0) implies to fix the factor correlation to 1.0 in M0. But I am definitely no expert on chi2-difference testing.
bmuthen posted on Friday, November 26, 2004 - 12:42 pm
Yes, fixing a factor correlation to 1 is a common approach to this. But that may cause the chi-square approximation to not be good. There is a general rule that 2 nested models do not give a proper chi-square distribution if the nested model has a parameter value on the border of the admissible parameter space, such as a unit factor correlation. How approximate it is in your case, I don't know (it calls for a monte carlo simulation study). If you want to report these results, you should mention this potential problem, and perhaps show the results as descriptive evidence not inferential evidence.
Anonymous posted on Thursday, May 05, 2005 - 7:43 am
I'm interested in this "value on the border" issue. Is there any reference about it? And, is there a reference of a montecarlo simulation of that same issue? I mean how good is the chi-square difference test when a restriction has a value on the border...
Dear Madam or Sir, I have a question concerning fit indices for CFA with categorical variables and 4 groups using WLS (20.000 subjects). When I set factor loadings and thresholds free in each group, CFI increases as expected, but RMSEA increases and TLI decreases. Should I ignore the latter two or does this indicate something else?
sample in/output follows. thank you very much Hage Wolff
****INPUT full invariance Data: ngroups = 4; ... Variable: names are i2 t1 t3 i4 t5 o4 o5 i6 o8; Categorical = all; Analysis: Type = mgroup meanstructure; Estimator = WLS; para =delta; Model: F1 by i2 i4 i6 t1 t3 t5 o4 o5 o8; ****OUTPUT Chi-Square Model Fit Value 5218.899/ df 210 Baseline Model Value 89035.852/ df 144 CFI 0.944 TLI 0.961 RMSEA 0.070
**** INPUT free factor loadings & thresholds Model: F1 by i2 i4 i6 t1 t3 t5 o4 o5 o8; Model group2: F1 by i2 i4 i6 t1 t3 t5 o4 o5 o8; F1 by i2@1; [i2$3 i2$4]; [i4$2 i4$3 i4$4]; [i6$2 i6$3 i6$4]; [t1$2 t1$3 t1$4]; [t3$2 t3$3 t3$4]; [t5$2 t5$3 t5$4]; [o4$2 o4$3 o4$4]; [o5$2 o5$3 o5$4]; [o8$2 o8$3 o8$4]; [similar for groups 3 & 4]
Chi-Square Model Fit Value 3720.447/ df 111 Baseline Model Value 89035.852 / df 144 CFI 0.959 TLI 0.947 RMSEA 0.082
The fit indices do not always change in tandem. This is clear in simulation studies such as Yu's 2002 dissertation which is on our web site under Papers. She found that CFI is often more dependable than other measures.