Jon Elhai posted on Sunday, December 03, 2006 - 12:37 pm
Dear Drs. Muthen, From reading the documentation and discussion emails, it sounds like the DIFFTEST command is to be used for WLSMV estimation when comparing two nested models.
I could not, however, find the answer to this question of mine... If I am comparing two models with DIFFTEST, how do I interpret the resulting p value? Does a statistically significant p value (e.g., < .05) merely mean that the less restrictive and more restrictive models are significantly different from eachother, without inferring directionality? If so, if DIFFTEST results in a statistically significant difference between models, would I merely examine the two models' goodness of fit indices, and assume that the model with the better fit was found by DIFFTEST to be statistically better?
I recall seeing one posting that suggested that a non-significant DIFFTEST merely means that the more restrictive model cannot be assumed to have a significantly poorer fit. This suggests to me that directionality is an issue. And if this is the case, I wonder how to test using DIFFTEST the hypothesis that the more restrictive model is significant better than the less restrictive model, in terms of fit.
I am using DIFFTEST to check if parameters can be equated across multiple groups. While I have gotten a significant result (p=0.0184) from the Chi-Square Test for Difference Testing, which I understand indicates that the unrestricted model (fit first, used to generate deriv.dat) fit better. However, the other fit statistics suggest that the constrained model fits relatively better (i.e. CFI & TLI are greater, RMSEA is lower for the restricted model). Which indicator (DIFFTEST versus CFI/TLI/RMSEA) should I use to choose the best model?
Chad Gundy posted on Thursday, December 03, 2009 - 2:42 am
Dear Drs. Muthen,
I have a question about testing nested models using the DIFFTEST function for WLSMV estimators.
I tried to directly compare several models which I had thought were nested in each other, and DIFFTEST had no complaints: everything seemed to work well.
However a colleague pointed out that one of my models did not seem to be nested in another one. Namely, both models were two dimensional CFA models, and the "nested" model was clearly more restricted, for it had two extra fixed parameters. However, in the two models, an observed variable was allowed to load on a different factor.
My colleague also objected to directly comparing a 1st order CFA with a higher order CFA.
My question is whether I would be justified in using DIFFTEST in these cases, noting that it doesn't complain about any problems? If so, how can I explain this to my colleague? If not, why not?
Mplus checks that the nested model, the more restrictive model, has a worse fitting-function values and fewer parameters than the other model. This does not totally insure that the model is nested.
I don't think the model with an observed variable loading on a different factor is nested. I think the other model is nested because it restricts the psi matrix but there may be something else I do not know which would make it not nested.
Catherine posted on Friday, February 25, 2011 - 8:22 am
Dear Drs Muthen,
I want to use the Difftest option to compair a 2factor model with the same model but with measurement errors allowed to correlate. But all i get is this: THE CHI-SQUARE DIFFERENCE TEST COULD NOT BE COMPUTED BECAUSE THE H0 MODEL IS NOT NESTED IN THE H1 MODEL.
Kurt Beron posted on Friday, April 15, 2011 - 12:23 pm
Dear Drs. Muthen,
I am running CFAs with categorical data and using DIFFTEST for my nested models. Things work fine when I use WLSMV. However I have some models with many parameters and receive the program's advice to try ULSMV with them given the extraordinarily long time for convergence otherwise. I've tried this and also used DIFFTEST with this based on the program output that says:
* The chi-square value for MLM, MLMV, MLR, ULSMV, WLSM and WLSMV cannot be used for chi-square difference testing in the regular way. MLM, MLR and WLSM chi-square difference testing is described on the Mplus website. MLMV, WLSMV, and ULSMV difference testing is done using the DIFFTEST option.
However when I run ULSMV with DIFFTEST I get the message:
*** WARNING in ANALYSIS command DIFFTEST is valid only for estimators WLSMV and MLMV. Request for DIFFTEST will be ignored.
I'm missing something here. The manual seems silent on ULSMV for this.
Would you explain what the proper difference test is to use here and how I should implement it? I am using v6.1.
Kurt Beron posted on Sunday, April 17, 2011 - 12:01 pm
I have 18 indicators for one latent variable and 8 for the second in one time period, and then I have the same setup for a different time period, and then I constrain across time periods. For example, the actual code for the constrained model is
Kurt Beron posted on Sunday, April 17, 2011 - 12:04 pm
And one addendum to the previous post is that this is my test file - which still works with wlsmv but doesn't with ulsmv. However the one that is the time consuming one has this over five time periods, not just two.
So with 5 time periods you have 10 factors and 130 categorical items. That's a tough model to fit in either WLSMV or ML (which is also available in Mplus). WLSMV takes a long time due to the large weight matrix for many variables and ML takes a long time due to the numerical integration over 10 dimensions. With ML, Monte Carlo integration could possibly be used but LRT testing is problematic with Monte Carlo due to only approximate loglikelihoods.
I don't think ULSMV helps here given that you need DIFFTEST. In version 6.1, ULSMV is inadvertently shut off in connection with DIFFTEST (which will be fixed in the new 6.11 version coming shortly), but my testing of a 72-item example shows that ULSMV isn't faster than WLSMV. This is because you can't use NOSERR and NOCHI since you need "TECH3-type" information for the second step of DIFFTEST.
I guess I would try WLSMV and not work with all 5 time points together in order to reduce the size of the problem.
Kurt Beron posted on Sunday, April 17, 2011 - 2:48 pm
Thanks, Bengt. I have worked on cutting the problem into pieces but wanted to make sure the DIFFTEST issue with ULSMV wasn't suggesting some other issue I needed to be aware of. With your information I'll keep going with the current splitting process and not worry about 6.11 fixing the "feature" of 6.1.
Jo Brown posted on Friday, June 01, 2012 - 4:00 am
I was planning to use the DIFFTEST option to estimate the difference in parameters between boys and girls in my sample.
However, the girls and boys files are separate as I ran multiple imputation on boys and girls separately.
Is there a way to still use DIFFTEST when the groups you want to compare are not in the same file or should I considerate alternatives?
I want to compare two nested models, but I was wondering whether the chi-square difference test using the WLSMV and MLMV estimators (DIFFTEST) is, just like regular chi-square test, dependent on sample size? Thanks in advance for your help! Ank
I am running a multigroup CFA with 4 binary indicators for one continuous factor using WLSMV. The goal is to compare nested models using the DIFFTEST option in order to identify measurement non-invariance.
I have established the configural invariance model as a baseline for the DIFFTEST using the model constraints described in the UG (Referent loading @1, all other loadings free, all thresholds free, all scaling matrices@1 and factor means@0).
Scalar invariance was rejected, so I estimated partial invariance models based on modification indices. When freeing the loading and threshold of a non-invariant item, I set its scaling factor to 1 according to the UG. For one non-invariant item the DIFFTEST option worked. The model fit was still not satisfactory, however. I therefore released the threshold and loading of another item, again setting its scaling factor to 1. When running the model, I receive the message that DIFFTEST could not be used because H0 is not nested in H1. I do not see how this is possible. As far as I can see the model is perfectly nested in the configural model. I am wondering if I need to set factor means to zero in this partial invariance model because I am releasing loadings and thresholds for half of my indicators.
Yvonne LEE posted on Wednesday, July 09, 2014 - 5:04 pm
I am new to Mplus and is running SEM with WLSMV estimation on a rather small offender sample (N = 175). In the end, the model chosen with fit indexes: RMSEA = 0.076; WRMR =0.773; CFI = 0.736; TFI = 0.705. I did not refer to the chi-square test because of the small sample. Would like to seek your expert advice on the followings:-
1. Based on RMSEA and WRMR, the model fit is acceptable. However, CFI and TFI is not good. Is that OK? I read some literature review suggests RMSEA <= 0.8 or <=0.6 and WRMR <=1 as the cut-off.
2. When compared to the alternative model, the DIFFTEST shows a significant result despite the less restrictive model shows RMSEA = 0.072. For a small sample research, is it proper to rely on DIFFTEST which is based on chi square test?
That is a poor CFI. Also, I think RMSEA is a little high - you want it to be at most 0.05. RMSEA is also based on chi-2. I would use chi-2 as a rough approximation - and also DIFFTEST in the same way. Seems like the model needs more work.
I want to compare a two-factor (less restrictive model) to a one-factor (more restrictive model) CFA using 10 binary indicators and WLSMV estimator. I am using the Add Health dataset.
On the two-factor model, 5 items load onto a nonviolent antisocial behavior factor and 5 items load onto a violent antisocial behavior factor. On the one-factor model, all 10 items load onto one factor of antisocial behavior.
I used the DIFFTEST option to compare the models, and there is only one degree of freedom for the Chi Square Test for Difference Testing. Is the one parameter difference because 1 factor variance parameter is estimated in the one-factor model instead of 2 factor variances in the two-factor model?
I ran a simple path model where I have two continuous variables predicting a binary outcome with several covariates. I used the WLSMV estimator. I ran a multigroup model to determine whether the model differs by sex using the DIFFTEST command, however, I am having some trouble with the interpretation. Comparing the less restrictive model (unconstrained) with the more restrictive model (constrained) results in a non-significant chi-square difference. Does this mean that the model does not differ by sex? Any help would be greatly appreciated!
I noticed on an earlier post on this thread by Jaime Derringer (posted on Tuesday, April 14, 2009 - 1:30 pm) you suggested that the p-value from the DIFFTEST results should be used for model selection over comparing the model fit indices (CFI/TLI & RMSEA), when comparing nested models using the WLSMV estimator.
I have a similar situation. The p-value from the DIFFTEST results suggests the constrained model fits WORSE, but the model fit indices suggest the constrained model fits BETTER.
I did what you suggested in a manuscript; however, a reviewer is confused by this. He says that I should go off the model fit indices and not the DIFFTEST results 1) because the model fit indices better take into account model parsimony and 2) because my sample size is very large (N = 7617) and the DIFFTEST results are highly influenced by sample size.
Is he right? If not, how would you recommend I reply to him?
I don't think I've seen a study of how changes in model fit indices performs well with categorical outcomes. Even if DIFFTEST is powerful at this n, you can see if the model changes that are indicated are substantively important.