Jon Elhai posted on Sunday, December 03, 2006 - 12:37 pm
Dear Drs. Muthen, From reading the documentation and discussion emails, it sounds like the DIFFTEST command is to be used for WLSMV estimation when comparing two nested models.
I could not, however, find the answer to this question of mine... If I am comparing two models with DIFFTEST, how do I interpret the resulting p value? Does a statistically significant p value (e.g., < .05) merely mean that the less restrictive and more restrictive models are significantly different from eachother, without inferring directionality? If so, if DIFFTEST results in a statistically significant difference between models, would I merely examine the two models' goodness of fit indices, and assume that the model with the better fit was found by DIFFTEST to be statistically better?
I recall seeing one posting that suggested that a non-significant DIFFTEST merely means that the more restrictive model cannot be assumed to have a significantly poorer fit. This suggests to me that directionality is an issue. And if this is the case, I wonder how to test using DIFFTEST the hypothesis that the more restrictive model is significant better than the less restrictive model, in terms of fit.
I am using DIFFTEST to check if parameters can be equated across multiple groups. While I have gotten a significant result (p=0.0184) from the Chi-Square Test for Difference Testing, which I understand indicates that the unrestricted model (fit first, used to generate deriv.dat) fit better. However, the other fit statistics suggest that the constrained model fits relatively better (i.e. CFI & TLI are greater, RMSEA is lower for the restricted model). Which indicator (DIFFTEST versus CFI/TLI/RMSEA) should I use to choose the best model?
Chad Gundy posted on Thursday, December 03, 2009 - 2:42 am
Dear Drs. Muthen,
I have a question about testing nested models using the DIFFTEST function for WLSMV estimators.
I tried to directly compare several models which I had thought were nested in each other, and DIFFTEST had no complaints: everything seemed to work well.
However a colleague pointed out that one of my models did not seem to be nested in another one. Namely, both models were two dimensional CFA models, and the "nested" model was clearly more restricted, for it had two extra fixed parameters. However, in the two models, an observed variable was allowed to load on a different factor.
My colleague also objected to directly comparing a 1st order CFA with a higher order CFA.
My question is whether I would be justified in using DIFFTEST in these cases, noting that it doesn't complain about any problems? If so, how can I explain this to my colleague? If not, why not?
Mplus checks that the nested model, the more restrictive model, has a worse fitting-function values and fewer parameters than the other model. This does not totally insure that the model is nested.
I don't think the model with an observed variable loading on a different factor is nested. I think the other model is nested because it restricts the psi matrix but there may be something else I do not know which would make it not nested.
Catherine posted on Friday, February 25, 2011 - 8:22 am
Dear Drs Muthen,
I want to use the Difftest option to compair a 2factor model with the same model but with measurement errors allowed to correlate. But all i get is this: THE CHI-SQUARE DIFFERENCE TEST COULD NOT BE COMPUTED BECAUSE THE H0 MODEL IS NOT NESTED IN THE H1 MODEL.
Kurt Beron posted on Friday, April 15, 2011 - 12:23 pm
Dear Drs. Muthen,
I am running CFAs with categorical data and using DIFFTEST for my nested models. Things work fine when I use WLSMV. However I have some models with many parameters and receive the program's advice to try ULSMV with them given the extraordinarily long time for convergence otherwise. I've tried this and also used DIFFTEST with this based on the program output that says:
* The chi-square value for MLM, MLMV, MLR, ULSMV, WLSM and WLSMV cannot be used for chi-square difference testing in the regular way. MLM, MLR and WLSM chi-square difference testing is described on the Mplus website. MLMV, WLSMV, and ULSMV difference testing is done using the DIFFTEST option.
However when I run ULSMV with DIFFTEST I get the message:
*** WARNING in ANALYSIS command DIFFTEST is valid only for estimators WLSMV and MLMV. Request for DIFFTEST will be ignored.
I'm missing something here. The manual seems silent on ULSMV for this.
Would you explain what the proper difference test is to use here and how I should implement it? I am using v6.1.
Kurt Beron posted on Sunday, April 17, 2011 - 12:01 pm
I have 18 indicators for one latent variable and 8 for the second in one time period, and then I have the same setup for a different time period, and then I constrain across time periods. For example, the actual code for the constrained model is
Kurt Beron posted on Sunday, April 17, 2011 - 12:04 pm
And one addendum to the previous post is that this is my test file - which still works with wlsmv but doesn't with ulsmv. However the one that is the time consuming one has this over five time periods, not just two.
So with 5 time periods you have 10 factors and 130 categorical items. That's a tough model to fit in either WLSMV or ML (which is also available in Mplus). WLSMV takes a long time due to the large weight matrix for many variables and ML takes a long time due to the numerical integration over 10 dimensions. With ML, Monte Carlo integration could possibly be used but LRT testing is problematic with Monte Carlo due to only approximate loglikelihoods.
I don't think ULSMV helps here given that you need DIFFTEST. In version 6.1, ULSMV is inadvertently shut off in connection with DIFFTEST (which will be fixed in the new 6.11 version coming shortly), but my testing of a 72-item example shows that ULSMV isn't faster than WLSMV. This is because you can't use NOSERR and NOCHI since you need "TECH3-type" information for the second step of DIFFTEST.
I guess I would try WLSMV and not work with all 5 time points together in order to reduce the size of the problem.
Kurt Beron posted on Sunday, April 17, 2011 - 2:48 pm
Thanks, Bengt. I have worked on cutting the problem into pieces but wanted to make sure the DIFFTEST issue with ULSMV wasn't suggesting some other issue I needed to be aware of. With your information I'll keep going with the current splitting process and not worry about 6.11 fixing the "feature" of 6.1.
Jo Brown posted on Friday, June 01, 2012 - 4:00 am
I was planning to use the DIFFTEST option to estimate the difference in parameters between boys and girls in my sample.
However, the girls and boys files are separate as I ran multiple imputation on boys and girls separately.
Is there a way to still use DIFFTEST when the groups you want to compare are not in the same file or should I considerate alternatives?
I want to compare two nested models, but I was wondering whether the chi-square difference test using the WLSMV and MLMV estimators (DIFFTEST) is, just like regular chi-square test, dependent on sample size? Thanks in advance for your help! Ank
I am running a multigroup CFA with 4 binary indicators for one continuous factor using WLSMV. The goal is to compare nested models using the DIFFTEST option in order to identify measurement non-invariance.
I have established the configural invariance model as a baseline for the DIFFTEST using the model constraints described in the UG (Referent loading @1, all other loadings free, all thresholds free, all scaling matrices@1 and factor means@0).
Scalar invariance was rejected, so I estimated partial invariance models based on modification indices. When freeing the loading and threshold of a non-invariant item, I set its scaling factor to 1 according to the UG. For one non-invariant item the DIFFTEST option worked. The model fit was still not satisfactory, however. I therefore released the threshold and loading of another item, again setting its scaling factor to 1. When running the model, I receive the message that DIFFTEST could not be used because H0 is not nested in H1. I do not see how this is possible. As far as I can see the model is perfectly nested in the configural model. I am wondering if I need to set factor means to zero in this partial invariance model because I am releasing loadings and thresholds for half of my indicators.
Yvonne LEE posted on Wednesday, July 09, 2014 - 5:04 pm
I am new to Mplus and is running SEM with WLSMV estimation on a rather small offender sample (N = 175). In the end, the model chosen with fit indexes: RMSEA = 0.076; WRMR =0.773; CFI = 0.736; TFI = 0.705. I did not refer to the chi-square test because of the small sample. Would like to seek your expert advice on the followings:-
1. Based on RMSEA and WRMR, the model fit is acceptable. However, CFI and TFI is not good. Is that OK? I read some literature review suggests RMSEA <= 0.8 or <=0.6 and WRMR <=1 as the cut-off.
2. When compared to the alternative model, the DIFFTEST shows a significant result despite the less restrictive model shows RMSEA = 0.072. For a small sample research, is it proper to rely on DIFFTEST which is based on chi square test?
That is a poor CFI. Also, I think RMSEA is a little high - you want it to be at most 0.05. RMSEA is also based on chi-2. I would use chi-2 as a rough approximation - and also DIFFTEST in the same way. Seems like the model needs more work.
I want to compare a two-factor (less restrictive model) to a one-factor (more restrictive model) CFA using 10 binary indicators and WLSMV estimator. I am using the Add Health dataset.
On the two-factor model, 5 items load onto a nonviolent antisocial behavior factor and 5 items load onto a violent antisocial behavior factor. On the one-factor model, all 10 items load onto one factor of antisocial behavior.
I used the DIFFTEST option to compare the models, and there is only one degree of freedom for the Chi Square Test for Difference Testing. Is the one parameter difference because 1 factor variance parameter is estimated in the one-factor model instead of 2 factor variances in the two-factor model?
I ran a simple path model where I have two continuous variables predicting a binary outcome with several covariates. I used the WLSMV estimator. I ran a multigroup model to determine whether the model differs by sex using the DIFFTEST command, however, I am having some trouble with the interpretation. Comparing the less restrictive model (unconstrained) with the more restrictive model (constrained) results in a non-significant chi-square difference. Does this mean that the model does not differ by sex? Any help would be greatly appreciated!
I noticed on an earlier post on this thread by Jaime Derringer (posted on Tuesday, April 14, 2009 - 1:30 pm) you suggested that the p-value from the DIFFTEST results should be used for model selection over comparing the model fit indices (CFI/TLI & RMSEA), when comparing nested models using the WLSMV estimator.
I have a similar situation. The p-value from the DIFFTEST results suggests the constrained model fits WORSE, but the model fit indices suggest the constrained model fits BETTER.
I did what you suggested in a manuscript; however, a reviewer is confused by this. He says that I should go off the model fit indices and not the DIFFTEST results 1) because the model fit indices better take into account model parsimony and 2) because my sample size is very large (N = 7617) and the DIFFTEST results are highly influenced by sample size.
Is he right? If not, how would you recommend I reply to him?
I don't think I've seen a study of how changes in model fit indices performs well with categorical outcomes. Even if DIFFTEST is powerful at this n, you can see if the model changes that are indicated are substantively important.
Kelly Minor posted on Thursday, August 18, 2016 - 4:13 pm
I am trying to run a multiple group analysis on nested data with a categorical outcome. I am reading conflicting suggestions on what to use to test for differences between groups: Satorra-Bentler, Wald, and DIFFTEST have all been in discussion. Please advise the correct test.
I am examining predictors (x1, x2, x3) of college enrollment (0=none, 1=delayed, 2=immediate) using WLSMV. I want to see if these differ based on SES quartile. My N=14,018, and the (condensed) input is :
CLUSTER is SCH_ID ; WEIGHT is F2BYWT; CATEGORICAL is college ; GROUPING is SESquart (1=FIRST, 2=SECOND, 3=THIRD, 4=FOURTH);
ANALYSIS: TYPE = COMPLEX
!For the first input ; MODEL: college ON x1 ; college ON x2; college ON x3;
college ON covariates;
!estimate variances to ensure no deletion ;
!REPEAT FOR MODEL SECOND, MODEL THIRD, MODEL FOURTH; ---------------------------- !For the second input file; MODEL: college ON x1 (1); college ON x2 (2); college ON x3 (3);
college ON covariates ;
!estimate variances to ensure no deletion ;
!REPEAT FOR MODEL SECOND, MODEL THIRD, MODEL FOURTH;
Any information on how to compare groups and simultaneously accommodate weighting, nesting, and categorial outcomes is appreciated.
Margarita posted on Friday, August 19, 2016 - 2:41 am
Dear Dr. Muthen,
I am a bit confused regarding the fit indices given by the H0 model during difftest.
I initially tested a model, e.g:
Model 1= Y on A B; with 123 free parameters and x2(471)=1684.571, RMSEA = .038 (.037-040), CFI = 947, TLI = 941.
Then I decided to check the impact of a covariate (Model 2: Y on A B Covariate;) and then to statistically compare the two models using DIFFTEST.
So Model 1 became the H0 model where I constrained the covariate to zero = Y on A B Covariate@0; with again 123 free parameters but different fit indices than Model 1: x2(504)=3014.608, RMSEA = .056 (.054-058), CFI = 862, TLI = 846.
I thought that Model 1 and H0 would have similar fit indices given that the additional DV is constrained to 0? Should the fit indices of H0 model be taken into consideration?
Which of the two models (H0 vs.H1) should then be reported in a paper?
For your model, multiple group analysis is obtained by using TYPE=MIXTURE and the CLASSES and KNOWNCLASS options. When all classes are known, it is the same as multiple group analysis.
Margarita posted on Monday, August 22, 2016 - 8:49 am
Thank you for your reply. The slope is stat. significant for the first time points (I am running a 3x3 cross-lag model).
I understand that to compare two models using chi square the two models need to be nested. Would it then be wrong to compare the fit of the initial model (Y on A B) to that of H1 model (Y on A B Cov) if one is not interested in the chi square difference?
I have seen both in published papers, and I am just trying to understand what it the best practice from a statistical point of view (i.e. comparing nested models using chi square vs. non-nested models without chi square)
Yes, that would be wrong with the WLSMV estimator. You can do it with ML because then only the DVs need to be the same.
Yoosun Chu posted on Sunday, August 06, 2017 - 7:56 pm
Hello, I would like to do DIFFTEST in my two models. I thought that the two models are in a nested relationship, but the output said it isn't. My H1 is 20 items and I assume four factors. My H0 is 18 items (a subset of 20 items of H1) and I assume three factors. I assume that the issue might be the different sample size. The N in H0 is smaller than the N in H1 due to the missing on all variables. Any advice would be appreciated. Thanks.
When conducting diff testing for hierarchical models, I am getting the following warning:
WARNING: THE LATENT VARIABLE COVARIANCE MATRIX (PSI) IS NOT POSITIVE DEFINITE. THIS COULD INDICATE A NEGATIVE VARIANCE/RESIDUAL VARIANCE FOR A LATENT VARIABLE, A CORRELATION GREATER OR EQUAL TO ONE BETWEEN TWO LATENT VARIABLES, OR A LINEAR DEPENDENCY AMONG MORE THAN TWO LATENT VARIABLES. CHECK THE TECH4 OUTPUT FOR MORE INFORMATION. PROBLEM INVOLVING VARIABLE ILIFANT.
I attempted to impose model constraints to avoid heywoods. Diff testing will not run with model constraints. I get the diff test to run without the model constraints. Can my diff testing results without the constraints be trusted?
See if TECH4 can tell you why you get the message. Typically you have to modify the model, sometimes with the help of Modindices. Note that the message isn't due to Difftesting per se but due to one of the two models being analyzed.
Hi, we are testing several models. In all cases we use WLSMV (ordinal variables). We have identified some models whith adequate fit according to some index (RMSEA, CFI, TLI), but they are not nested. So, how we can compare them? In all cases chi2 is significant (we have samples >350). thanks!
I wanted to perform a diff test comparing a 6-factor model with a two bi-factor model. Is this possible? I wasn't sure if these two are nested models, but when I tried this anyways, Mplus output does say these are not nested models. Thanks.
Try our NESTED option. We have a section on this in our new paper on our website:
Asparouhov, T. & Muthén, B. (2018). Nesting and equivalence testing for structural equation models. Structural Equation Modeling: A Multidisciplinary Journal. DOI:10.1080/10705511.2018.1513795 (Download scripts).
I have compared several alternative models to one main model, using DIFFTEST. Only one of the alternative models had a better fit according to the DIFFTEST. However, all the fit indices (RMSEA, CFI, TLI, Chi2) are better in all alternative models. Although I read on the forum that I should rely on the DIFFTEST, I am not sure how to interpret it. Can you give some advice on how to explain why all the alternative models had better fit indices than the main model, but only one of these alternative models had a better fit based on the DIFFTEST?
RMSEA, CFI, TLI don't have statistical significance attached so even though they are better we can't say that they are statistically significantly better. Chi2 is better by definition, i.e., it is always better,i.e., if the models are nested the chi-square will be better for the less restricted model. Statistical significance is obtained by the DIFFTEST command. You can also use the model test to test between the various models as an alternative.
So, the models improve, but not enough to be a real improvement? Is there also a role for the number of parameters in the models? As less parameters is usually preferred, the relation between model fit and number of parameters does not improve enough to prefer the simpeler model?
Yes. The DIFFTEST has a number of parameter penalty imbedded in it. Just like the standard LRT - the test statistic is compared to a chi-square distribution with degrees of freedom that is equal to the excess of the number of parameters.
Im using DIFFTEST to compare nested models, to see if a less restrictive model (grouping by sex) is better. In my H1 model, is it ok to unrestrict each path one at a time to make a sort of interative comparison? Or should i unrestrict all paths at the same time to compare with the the H0 model where all paths are restricted?
When i compare H1 with all paths unrestricted to H0, DIFFTEST is non sig. But when i unrestrict certain paths but leave others restricted in H1 and compare with fully restricted H0 i get a sig DIFFTEST, suggesting there are sig differences between men and women for some paths but not for others.