Message/Author 

Sanjoy posted on Wednesday, April 13, 2005  5:52 pm



Dear Professor Muthen/s Before asking the main questions related to "difftest" let me clarify couple of things. Please rectify me if I'm wrong. 1. I have MPlus version (3.12). I suppose this one is the most updated. I could not find the example 12.12 (page 278) in the "User's Guide example folder", in fact for that matter there is not a single file from chapter 12 of the MPlus User's Guide Manual in the above mentioned folder that we got from MPlus CD 2. As an alternative therefore, I replicate ur codes (page 278) and tried ... it fails to run, it said " *** FATAL ERROR VARIABLE Y7 CAUSES A SINGULAR WEIGHT MATRIX PART. THIS MAY BE DUE TO THE VARIABLE BEING DICHOTOMOUS BUT DECLARED AS CONTINUOUS. RESPECIFY THE VARIABLE AS CATEGORICAL." … same for the variables Y8 and Y9 Next, I include y7y9 as categorical in the command line and tried to run ... THEN it WORKED WELL, I have used the same data set that u have mentioned at page 278, though u have NOT mentioned y7y9 as categorical ... I just need to make sure I have not messed up, kindly rectify me if I have done so Now, coming to the DIFFTEST issues Q1. Usually, though not always, under Null Hypothesis (H0) we assume less restrictive model and under Alternative Hypothesis (H1) we put the restrictions on model parameters (e.g. chow test) ...it looks here for the case of "difftest" we alter the usual practices, why is it so Q2. Can you suggest some article written on “Difftest” practice Q3. How should we use the “Difftest” result? From the second step result I got this “ ChiSquare Test for Difference Testing Value 2.968 Degrees of Freedom 3** PValue 0.3953 ” Usually a pvalue close to zero (0.05 is a typical threshold) signals that our null hypothesis is false and we reject Null while a Large pvalues (like the above .39) implies that there is no detectable difference for the sample size used, and therefore we fail to reject Null. … However in this “difftest” case, would it be the reverse ? Thanks and regards 

BMuthen posted on Wednesday, April 13, 2005  11:24 pm



1. The examples from Chapter 12 are not included with the Mplus CD. 2. I would have to see the full model to answer this question. A pvalue greater than .05 says that the restrictions cannot be rejected that is the restrictions do not worsen the fit of the model. There is currently no article written on DIFFTEST. 

Sanjoy posted on Thursday, April 14, 2005  9:37 am



Thank you Professor ... I will mail you the full model 

Sanjoy posted on Thursday, April 14, 2005  10:47 am



Dear Professor .... Why it's the case that for WLSMV the conventional approach of taking the difference between the chisquare values and the difference in the degrees of freedom is not appropriate ... I mean Q1. How can we show that the standard chisquare difference is NOT distributed as chisquare and Q2. How do ensure that DIFFTEST is doing the correct thing Thanks and regards 

BMuthen posted on Friday, April 15, 2005  1:29 am



You may want to look at the literature by Satorra and Bentler on robust chisquare difference testing with continuouos nonnormal outcomes. The issues are the same. You can do a simulation study to see how well DIFFTEST performs. There will be a forthcoming paper on the DIFFTEST theory. 

Sanjoy posted on Friday, April 15, 2005  9:17 am



1. Is it the article "A scaled difference chisquare test statistic for moment structure analysis", Psychometrika, 66,507514, 2001(A. Satora and P.M. Bentler) that you have referred me to check with ... or something else 2. I'm severely time constrained, nonetheless I will try the simulation things...in between, if you kindly send me an electronic copy of "forthcoming paper on the DIFFTEST theory" ... that would be a tremendous help to me ... if the authors prohibit us from quoting, it's goes without saying that we will stick to that, however reading their article will help me to understand the nuances of DIFFTEST more comprehensively Thanks and regards 

BMuthen posted on Saturday, April 16, 2005  4:25 am



1. Yes. 2. The paper is not ready to be sent at this time. 


Dr. Muthen, I am trying to follow example 12.12 in the Mplus version 4 manual to use the chisquare difference test in models involving the WLSMV estimator. I am receiving the following error message: THE CHISQUARE DIFFERENCE TEST COULD NOT BE COMPUTED BECAUSE THE H0 MODEL IS NOT NESTED IN THE H1 MODEL. My second step model involves restraining 2 regression coefficients as being equal: y1 ON x1 (1); y1 ON x2 (1); These predictors were freely regressed on y1 in the model that I am using in the first step as indicated on p. 314. My interest is to test whether fixing these regression coefficients in the second step deteriorates the model fit. Is this possible to do following example 12.12, or am I off basis with regard to using the chisquare difference test for such a purpose? Thank you. 


It sounds like what you are doing is possible. You would need to send your input, data, output, and license number to support@statmodel.com for us to say any more. 

D C posted on Friday, September 24, 2010  2:48 pm



Hello Professors, I am doing a multiple group analysis of a factor structure defined by categoricalordinal indicators. I am using the WLSMV estimator, and hence I use the DIFFTEST to judge whether various restrictions imposed on the model significantly worsen the fit. However, my data has a relative large sample size (N=3650 with 2100 in one group category and the 1550 in the other. My questions are: 1. Is the DIFFTEST sensitive to large sample size as the CHISquare tests? 2. If so, I would like to use differences in CFIs (Meade et al, 2008)values to help judge the difference in model fit between restricted and less restricted models. So, in multiple group analysis (when GROUP statement is used)and various restrictions are imposed on a series of models, are the CFI values estimated each time anew? That is, is it advisable to take differences of CFI values between the restricted and less restricted models to judge model fit? Thank you! 


1. Somewhat less sensitive. 2. Each model has its own CFI. I don't believe using CFI to compare models is well established. 


Dear Dr. Muthen, I am running a structural equation model with categorical latent variables (both IV and DV) and I am attempting to do a ChiSquare difference test for my multiple group analysis. I have been running into problems when attempting to do the twostep chisquare test of model fit required when using a WLSMV estimator. After saving my derivatives in stepone, with every pathway constrained, I then go on to unconstrain one pathway for one group. Here is where I run into problems. I keep getting the error message : THE CHISQUARE DIFFERENCE TEST COULD NOT BE COMPUTED BECAUSE THE H0 MODEL IS NOT NESTED IN THE H1 MODEL. The chisquare in the second model is larger and there are a greater number of degrees of freedom (the model with one pathway unconstrained), compared to the baseline, fully constrained model. Is there something I’m not doing properly that is making my H0 not nest in the H1 model? Any guidance you can give me would be much appreciated. I can send you my input and data if that is helpful. Thank you! 


Please send the two outputs and your license number to support@statmodel.com. 


Dear Dr. Muthen, 2 questions: If with WLSMV, the Chi² values cannot be interpreted, and DIFFTEST must be used to compare nested models, then 1. am I correct in assuming that this implies also, that the other Fit Indices which are based on Chi² (such as RMSEA, CFI etc.) can NOT be interpreted? 2. a significant DIFFTEST is unidirectional and ALWAYS suggests that the more restricted Model has a worse fit? Thank you very much, Sabine 


1. All fit statistics can be interpreted. 2. Yes. 


Dear Dr Muthen I am fitting a cross lagged model and comparing group differences among white, hispanic and black. i am using categorical indicators for my latent factors. The difftest comparing the invariance and non invariance is ChiSquare Test for Difference Testing Value 103.284 Degrees of Freedom 38 PValue 0.0000 but RMSEA (0.04) TLI(.982) and CFI (.98) of the more restricted model are better than the RMSE (0.051) TLI (0.978) and CFI (0.966) of the less restricted model. should not it be otherwise? i mean if the diffest is significant then should i expect that the goodness of fit of the less restricted model were better than the indicators of the more restricted model? thank you fernando 


I don't know why RMSEA and CFI are so good for the more restrictive model, but I assume that the chi2 is bad also for the less restrictive model. In such cases, these fit indices don't always come out in an expected order of magnitude. I would rely more on the chi2 DIFFTEST. 


thank you very much, the chi2 for the less restricted model is 1079.328* while the chi 2 for the more restricted model is 911.407* would you know of some literature i could cite to support what you recommend? 


No, I think this is an open research area. 


Dear Dr. Muthén, my question concerns the interpretation of the chisquare difference test under WLSMV estimator. In your post above (April 13, 2005 at 11:24pm) you suggested that "A pvalue greater than .05 says that the restrictions cannot be rejected that is the restrictions do not worsen the fit of the model". However, on the UCLA website (precisely here: http://www.ats.ucla.edu/stat/mplus/faq/difftest.htm) an opposite interpretation of the pvalue seems to be followed. Am I missing something? I cannot see how the two interpretations could match. Please, I will be grateful if you would suggest me some key references as well. Many thanks! 


I think they are saying the same thing in a slightly different way. See DIFFTEST under Technical Appendices on the website. 

Suhaer Yunus posted on Thursday, November 28, 2013  12:27 pm



The independent variables in my study are binary and I have computed EFA (using Mplus version 7.1). The EFA results show that there are four first order correlated factors. The CFA results confirm that too. Now I want to show that whether four correlated factors model is better or whether there should be one higher order factor representing the four first order factors or a single factor measuring all items that form four factors. I understand that the models are not really nested so the DIFFTEST option may not be appropriate. I have computed three different models separately but how can I compare the results of these to choose the best one. Can I report the change in Chisquare and change df for these results? The results of the models are: Base Model  Distinct First Order Factors Chsq  1160.660*(df= 48) RMSEA= 0.034 CFI=0.935 TLI= 0.911 Model A –Second Order Model ChiSq (1106.889*)(df=50) RMSEA= 0.032 CFI= 0.938 TLI=0.919 But it suggest 0.000 correlation for one first order factor with the higher order factor. Model B  Single Factor ChiSq=9593.979*(df= 54) RMSEA=0.093 CFI=0.444 TLI=0.321 


I believe the secondorder factor model is nested in the firstorder correlated factor model. I would use DIFFTEST. 


Hi Linda, Thanks for your reply. I have computed the DIFFTEST. The four correlated factor model is the least restrictive. The second order is more restrictive and the single factor is most restrictive. I am comparing the single factor to the second order factor. With ESTIMATOR=WLSMV and PARAMETRIZATION=DELTA I get the following results: Base Model  First Order Chsq  1160.660*(df= 48) RMSEA= 0.034 CFI=0.935 TLI= 0.911 Second Order Model ChiSq (1520.644*)(df=50) Chisquare test for difference testing: Value= 223.535 Df=2 pvalue= 0.000 RMSEA= 0.038 CFI= 0.914 TLI=0.887 Single Factor ChiSq=9280.721*(df= 54) Chisquare test for difference testing: Value=6388.571 Df=(4) Pvalue= 0.000 RMSEA=0.091 CFI=0.462 TLI=0.343 I have following queries: 1If I have used the correct PARAMETRIZATION. If I use PARAMETRIZATION= THETA I get considerably low fit indices on all my models i.e. For Base model: Chsq  1796.660*(df= 48) RMSEA= 0.042 CFI=0.898 TLI= 0.860 and WORSE for other models 2 Single factor be compared with four correlated factors instead? 3All pvalues in chisquare DIFFTEST are greater than 0.5. Does it mean that the restrictions worsen the fit of the data. Thanks. 


Typically the H1 model is the least restrictive model. See the DIFFTEST technical appendix on the website. You should update to the current version of Mplus. 

SY Khan posted on Monday, March 24, 2014  6:04 am



Dear Dr. Muthen, I am trying to see if a higher order factor for my indepndent variables is better than four correlated factors through DIFFTEST. Only a few of my factor indicators are binary and the rest are composite variables depicting a range. I am getting the following message. THE MODEL ESTIMATION TERMINATED NORMALLY THE CHISQUARE DIFFERENCE TEST COULD NOT BE COMPUTED BECAUSE THE H0 MODEL MAY NOT BE NESTED IN THE H1 MODEL. DECREASING THE CONVERGENCE OPTION MAY RESOLVE THIS PROBLEM. I have tried different convergence values: 1 default 2 CONVERGENCE=0.05; 3CONVERGENGENCE= 0.000001; values 13 don't work and I get the same error message.In the output I have noticed that there are two values for CONVERGENCE CRITERION i.e. convergence criterion and another convergence criterions for H1. Which value needs reducing? If I set CONVERGENCE= 0.15 or a value higher than 0.15 it gives me the DIFFTEST results. But I am not sure if it is ok to use CONVERGENCE=0.15 or above. Please advise. Thanks for you guidance in advance. 


Please send the two outputs and your license number to support@statmodel.com so I can see why the models are not nested. Changing the convergence criteria will not solve an identification problem. 


Hi there, I conducted a chisquare difference test for 2 fairly complex nested models with categorical variables using WLSMV and found a stat sig result that I expected. BUT, there were several error term correlations specified through the CFA that I could not include in this analysis because it could not converge. I had previously run both of the models with the specifications included and written up the analysis. I wanted to add in this chi square diff test. Does it make sense to report the results of the chi square diff test? Please let me know what your recommendations are. Thank you! 


I would look into the reason for the nonconvergence. If the difference test is not on the models for which you reported results, I would not report it. 

Back to top 