Message/Author 


I have a copy of the instructions from your web site for chisquare difference testing using the SatorraBentler Scaled ChiSquare. I would like to perform such tests with nested models in the context of CFA. I am currently using EQS version 6.0. With this version, the robust estimation option provides the SB Scaled ChiSquare along with the robust CFI and the robust RMSEA measures of fit. This is my question. Is the MLM procedure in MPlus 2 (maximum likelihood parameter estimates with robust standard errors and a meanadjusted chisquare test statistic) the same estimation procedure as the robust procedure in EQS 6.0? If this is so, then I can follow the instructions in the paper on your WEB site to calculate the correct values for the SB Scaled Chi Square for comparing nested models. Thanks for your feedback. Len Burns 


MLM is the SatorraBentler chisquare test statistic, so it stands to reason that the directions would work for the EQS SatorraBentler chisquare as well as MLM. 

Wim Beyers posted on Monday, September 30, 2002  7:15 am



Hi there, I'm just asking a question here, because I did not receive any helpfull feedback on SEMNET. And here I find a specific forum on the topic, so... I calculated the SatorraBentler Scaled ChiSquare Difference Test by hand several times, but I sometimes come out with results that seem strange to me... For instance (n = 600): Comparison Model (79 df):  nonscaled chisquare = 249.18  scaled chisquare = 220.42 Nested Model (82 df):  nonscaled chisquare = 337.18  scaled chisquare = 308.24 So, a traditional chisquare diff test results favours the Comparison Model, with chisquarediff = 88.00 (df = 3). OK A SBS scaled diff test also favours the comparison Model, but following the calculations on http://www.statmodel.com/chidiff.html, the testvalue is 675.14. It's so huge I really hesitate to report it. Am I doing something wrong? Thanks for all help,  Wim Beyers Belgium 

bmuthen posted on Monday, September 30, 2002  9:42 am



It looks like you have done the calculations correctly according to our web site. The test value of 675 is certainly much larger than 88, but the p values are both very small and therefore similar so this could all be ok. Two other thoughts: You might have run into a local optimum in one of your 4 runs. You can run this in Mplus to get the scaling correction factors directly to see if they agree. 

Kaja LeWinn posted on Thursday, June 23, 2005  3:05 pm



Hello, We are trying to use the SatorraBenter scaled chisquared test using the instructions on this site. We have found that when we run our comparison model in MLM and ML we get different degrees of freedom (the same model, using two different estimation techniques gives us two different dfs). This does not happen in the nested model. We are confused by this. One characteristic of our model that might be of significance is that we are looking for measurement invariance using the grouping command. We were hoping this could be explained to us, as well as how we should approach the equation under point 3 of the online instructions (i.e. which degrees of freedom to we choose, those from the ML or MLM model). Thanks for your help, Kaja 


You need to send your outputs, data, and license number to support@statmodel.com. What you are describing has not been seen by us. There may be something else going on. 


Dear colleagues, I need to test out differences between nested models, and wonder if I can use the SatorraBentler scaled chi square in this particular case: I have a model with a set of predictors that are all regarded as metric (although a couple of them are dichotomies). Some of them could be analysed as latent variables, but I have chosen to start out using simple sumscores. It is therefore simply a path analysis that I am doing. I have two outcome variables, the most dependent one is a dichotomy. The other one, which is also regarded as a mediator, is an ordered categorical variable. Since the sample is based on clusters, I am using the cluster option in order to adjust for the design effect. I understand that the SB formula is based on ML and MLM estimators. The programme, however, does not permit ML and MLM estimators with this analysis. It only provides WLSMV estimation. Can my commands be changed in such a way that it allows for ML and MLM estimation? Or is there an alternative procedure that can be used for testing the difference between models? Most grateful for any suggestions. Leif 


MLM is the SatorraBentler chisquare. It is available only when all outcomes are continuous. 


Thanks for quick response. How do I test differences between nested path analysis models when I have categorical outcome variables and clustered data? Is this described in the Mplus manual or anywhere else? If such testing is not possible: What procedure would you recommend for identifying a "good" model? Leif 


In Chapter 15 under the ESTIMATOR option, there is a table that shows the estimators available for TYPE=COMPLEX and TYPE=TWOLEVEL. I'm not sure which you want to use for your clustered data. With the WLS estimator, difference testing is done in the usual way. With WLSM and MLR, you need to use a scaling correction factor which is given in the output and how to use it is shown on the website. With WLSMV, use the DIFFTEST option. See a description in the Mplus User's Guide. When you do not obtain chisquare, you can use 2 times the loglikelihood difference. 


I'm performing a CFA with type=complex as independency assumption is violated in my data set [gathered within families]. I used the SatorraBentler scales chisquare corrections for MLR as described on the Mplus website. My restricted model has an unscaled chisquare of 19.696 with df=13, and scaling correction factor of 1.123. My less restricted model has an unscaled chisqaure of 10.704 with df=12, and scaling correction factor of 1.213. Calculating the diff test scaling correction: (13*1.12312*1.213)/1=.043 the corrected chisq diff test would then be: (19.69610.704)/.043=209.116 I do not think this is correct. Can someone help me out? Thanks Sophie 

bmuthen posted on Friday, February 03, 2006  9:47 am



It looks like you have done this correctly. The asymptotics of this correction does not always work out in small samples as has been noted by the authors, although note that the p value will be zero for both uncorrected and corrected chisquare difference testing. In Mplus Version 4 you will also have access to a Wald test which avoids these problems. 


Thank you for the swift response. I'll see if I can lay my hands on Mplus 4. I presume however that I can also report the CFI, RMSEA, standardized residuals etc. to substantiate the improvement in model fit? Kinds regards Sophie 


Once you have computed the SatorraBentler scaled chisquare difference test, how can you determine whether the two models are significantly different or not? Thanks, Anna 


The other fits measures should be okay. 


You compare it to the chisquare table value for the number of degrees of freedom in the difference test. 


Thank you! Anna 


I've done a series of these calculations and I get some CD values that are negative (do*co is less than d1*c1). What should I do in this case? Use the absolute value of the CD? 


In this case, the difference testing is not working and should not be interpreted. 


What is my option here? Just a straight test based on the ML values? Also, to be clear  if the test is not working  is that attributable to some data characteristics or user error? I'm confident I'm doing the test correctly (as is comes out positive for about 50% of my comparisons). Thanks! 

bmuthen posted on Monday, February 20, 2006  6:30 pm



The asymptotics of the test fails to kick in  this has been observed among the creators of the test (SatorraBentler) in several applications. No user error, and no fault of the data (apart from perhaps not having a large enough n). An alternative is to use a Wald test which is part of the upcoming Mplus Version 4 (see new announcement on the home page). This test is robust to the same type of violations as MLR/MLM. 


Would the Wald result here be similar to what EQS calls the Lagrange Multiplier test? For example if I were to take my model and constrain a covariance to one  would the Lagrange value associated with freeing that parameter give me a test of the difference between a comparison model and a model with that constraint (given that is the only constraint I'm testing? Thanks again  great comments. 

bmuthen posted on Tuesday, February 21, 2006  3:16 pm



No, Wald testing pertains to restrictions on a given H0 model, whereas LM tests (= Mod Ind) pertains to relaxing restrictions on a given H0 model. Yes. The Wald test, however, would make it unnecessary for you to run that second run with your covariance restricted. 

jennybr posted on Monday, February 27, 2006  6:56 pm



HI, I want to compute a chisquare difference test, however, my two df's are the same. Does this mean that my models are not nested? 


What estimator are you using? 


There was some discussion above about the conditions under which the SatorraBentler chisquare test fails. Could you direct me to papers that discuss this further. Thanks, Anna 


I think Peter Bentler has written about this. I don't know of the exact references. If you can't find it doing a literature search, you can contact Peter Bentler. 


I have run a multigroup (4 groups) CFA using LISREL. The output indicates that the factor loadings are equivalent across the four groups  only one set of loadings in the output. However, a chisquare difference test (SantorraBentler Chisquare) is significant  contrained model minus unconstrained model. Does this sometimes happen? If so, what is the explanation and how does one proceed? Thanks Henry 


This would indicate that constraining the factor loadings to be equal across groups significantly worsens the fit of the model. There must be some factor loadings that are not invariant across groups. You can read about testing for measurement invariance in Chapter 13 of the Mplus User's Guide which is on the website. It is at the end of the multiple group discussion. 


I am searching for the formula for calculating degrees of freedom when using WLSMV, but cannot find the correct info in the latest User's Guide or in the Technical Appendices. I am using MPlus version 3.0 for this CFA. Can I interpret the degrees of freedom as accurate in the output? 


It is formula 110 in Technical Appendix 4 on the website. The only interpretable value for WLSMV is the pvalue for chisquare. The degrees of freedom are not calculated in the regular way. Difference testing of nested models can be carried out using the DIFFTEST option. 


I know that when using the SatorraBentler chisquare (MLM), you must calculate a corrected chisquare difference test. It appears from the discussion above that this SAME procedure should be used for the YuanBentler T2 test statistic (MLR). Is this true, or when using MLR (YuanBentler T2 test statistic) is it possible to just use a regular difference test? Thank you for your help! 


MLM, MLR, and WLSM all need to use a scaling correction factor for difference testing. 


Dear colleagues, I would like to use the Chisquare difference test but I am unsure if my two models are nested. I would like to compare a CFA with conflict as 1 scale (items 1,2,3,4,5,6,7,8) and a CFA with conflict as 2 scales( task conflict items 3,6,7 and relationship conflict items 1,2,5). The last model has 1 df less. Could I consider these models nested? If so could I use the SatorraBentler Scaled Difference Test? Thanks so much! Maria P.S. great forum, I have learned a lot from it! 


If you don't have the same set of observed dependent variables, the models are not nested. 


Thanks for your quick reply. I think I understand. So if I would make a second order factor (conflict) of the two first order factors (task conflict and relationship conflict)and compare this to a CFA of one factor (conflict) the models would be nested and thus I can use the SB Scaled Difference Test? Maria 


A secondorder factor model with two indicators is not identified. To be nested, each model would need to include the same set of dependent variables. I don't think if identified, your suggestion has that. 


I ran a multilevel CFA with continous variables. I would like to compare two models: (1) 2factors within and 1 factor between vs. (2) 2factors within and 2 factors between. As much as I understand Model 2 is not nested within model 1, so I can't use the SB Scaled Difference Test. Regarding my chisquare value and the fit indices I cannot decide which model fits better, because they are nearly the same. My question: Is there another way to test if they are different? If not, can I conclude both models are equivalent although I couldn't really prove it? Thanks very much! Janine 


BIC could be helpful to balance parsimony against improving the loglikelihood. If the fit information is about the same, I would decide based on the interpretation and usefulness of the model; I wouldn't be so concerned about testing. If the model with 2 between factors doesn't add interpretational value, parsimony speaks for the 1factor model on between. 


Thank you very much for your advice  very helpful! Janine 

Sunny Liu posted on Monday, March 09, 2009  12:28 am



I am a little confused with the SatorraBentler Scaled ChiSquare. For example, Baseline model: Chisquare Test of Model Fit: Value: 2 Degree of Freedom: 2 Scaling Correction Factor for MLR: 2 Model 1: Chisquare Test of Model Fit: Value: 1 Degree of Freedom: 1 Scaling Correction Factor for MLR: 1 Then # Compute the difference test scaling correction where d0 is the degrees of freedom in the nested model and d1 is the degrees of freedom in the comparison model. cd = (d0 * c0  d1*c1)/(d0  d1) = (2*2  1*1)/(21) = 3 Note that the ML chisquare is equal to the MLM or MLR chisquare times the scaling correction factor. # Compute the SatorraBentler scaled chisquare difference test (TRd) as follows: TRd = (T0  T1)/cd = (2*21*1)/3 = 1 The Xsquare difference test is Xsquare equal to 1 with 1 df, then it is not significant. Or should it be TRd = (T0  T1)/cd = (21)/3 = 1/3 The Xsquare difference test is Xsquare equal to 1/3 with 1 df, then it is not significant. Which way is correct? 


Is the SB chi^2 in Mplus going to be changed according to latest estimation provided by Satorra nd Bentler (2009)? see e.g., http://www.springerlink.com/content/k716217434q71737/fulltext.pdf Thanks Andrea 


We have on our list to include that twist in our next version. 


Dear Drs. Muthen, I'm doing a chisquare difference test for MLR. I get a negative cdvalue (=.016). I think this is due to the complex model (M1: df=1085 M2: df=1086, M1: Scale Cor=1.070 M2:1.069). It follows that also the scaled chisquare difference is negative (240,639). Is this result interpretable? How should I interpret it? Do I have to use the modulus of cd? Or does it mean, that the model with df=1086 fits better?? best regards Christoph Weber 


The result is not interpretable. This is a flaw with the method. 


I have the same problems with MLRtesting as described above (negative chisquare). In this situation, is it o.k. to use the Waldtest in combination with MLR and COMPLEX/CLUSTER in order to test some simple 1df constraints? 


Yes. 


Dear Drs. Muthen, I am trying the formula for positive SBdifference statistic values at invariance testing across groups (3). since the method requires estimating a M10 model, with the M0 output as starting values, I do not know which starting values to use in case of several groups. can you help me further, please? Cheers, kathrin 


You want to study examples 3 and 4 and their Mplus scripts at http://www.statmodel.com/examples/webnote.shtml#web12 You would use the M0 output for each group to form the multigroup M10 start values. 


Thank you for the fast response! I will check those notes. Cheers, kathrin 


Hi there  I am trying to run a fully unconstrained multiple group CFA model to compare to my constrained model, but I am not able to figure out the syntax for this. For the constrained model, I am using the syntax below. What do I need to add to make it unconstrained? Thank you! Title: CFA; low.hi; current; King DATA: FILE IS low.hi.current.csv; VARIABLE: NAMES ARE b1b5 c1c2 c3c7 d1d5 g; CATEGORICAL ARE b1b5 c1c2 c3c7 d1d5; GROUPING IS g (1 = low 2 = high); MODEL: f1 BY b1b5; f2 BY c1c2; f3 BY c3c7; f4 BY d1d5; 


See Slides 169 and 170 of the Topic 2 course handout on the website. 


Thank you for your prompt response! I realize this is probably a really easy question, but the syntax I posted above  is this for a fully unconstrained model and then syntax on slides 169 and 170 for a fully constrained model? Thank you so much! 


The syntax above is for a model with factor loadings and intercepts constrained to be equal across groups. This is the Mplus default. See Chapter 14 of the user's guide for a full description of multiple group analysis in Mplus. On Slides 169 and 170, the first syntax is for a fully constrained model. The second is for an unconstrained model. And the third is for a partially constrained model. 


Thanks again. We got a bit closer, but now are running into this problem  When the fully unconstrained model is run, we're getting error messages like this: THE MODEL ESTIMATION TERMINATED NORMALLY THE STANDARD ERRORS OF THE MODEL PARAMETER ESTIMATES COULD NOT BE COMPUTED. THE MODEL MAY NOT BE IDENTIFIED. CHECK YOUR MODEL. PROBLEM INVOLVING PARAMETER 81. THE CONDITION NUMBER IS 0.244D17. We're not able to get fit statistics because the parameter estimates couldn't be computed. When I check the output, I can find out that, in this model, parameter 81 is d5. So then I delete the line of code freeing d5 the model will run. I'm not sure what the condition number is yet. Can you tell us what the condition number indicates? Thanks! 


The condition number is related to identification. Please send the full output and your license number to support@statmodel.com. 

Jak posted on Tuesday, November 02, 2010  4:02 am



Dear Linda or Bengt, Is the method described in Webnote 12 ("Computing the Strictly Positive SatorraBentler ChiSquare Test in Mplus") also applicable to the YuanBentler T2 statistic obtained when using MLR estimation? Thanks in advance, Suzanne 


Yes. 

Ian Clara posted on Tuesday, December 07, 2010  6:39 am



Good morning. I am trying to conduct a chisquare difference test using the WLSMV estimation and type=complex. I have used the difftest option and for the two models that I want to test it says that they are not nested (although there is a single path that is different  set to zero in one model and freely estimated in the other). I wanted to try to conduct the chisquare difference test by hand, but I can't determine how to get the required output in Mplus v5. How can I obtain the scaling correction factors, or the log likelihood? Warm regards, Ian 


Difference testing using WLSMV cannot be done by hand. If you use WLSM, you will obtain a scaling correction factor. 

Fred Mueller posted on Wednesday, February 22, 2012  6:08 am



Dear Linda and Bengt, I would like to compare two nested models and I am using MLR as an estimator. Instead of using the (SantorraBentler Scaled) ChiSquare difference test, I would like to use the small difference in fit test by MacCallum, Browne, and Cai (2006, Psych Methods). For this test, I also have to indicate the ChiSquare of both models. How do I have to proceed? Can I just multiply the (regularI ChiSquare with the scaling correction factor to get the correct ChiSquare or is it more complicated? Thank you very much in advance! Best, Fred 


I don't know if the MacCallum et al approach needs the usual, uncorrected, chisquare or can use the MLR chisquare. You may want to approach the authors with that question. 

Fred Mueller posted on Wednesday, February 22, 2012  5:22 pm



Thank you very much for your quick reply! 


Dear Mplus Team, i am comparing 2 measurement models for 18 items using MLR. It would be great if you could help me with some answers: The first model is a onefactor model where all items are explaining by the same factor and the second model is a three factor model where each factor explains 6 items. (1)Is it right to say that these models are nested as the 3factormodel with the correlations between the factors set to 1 would equal the 1factormodel? (2) When i compute the SB scaled Chi square difference test, the result is negative. Therefore i tried computing the strictly positive SB scaled Chi square difference test. Thus i tried to estimate the more strict M0 model first which would be the 3factormodel with betweenfactorcorrelations set to one. However, the estimation did not converge. What am i doing wrong? Heres the input: MODEL: factor1 by f10f16; factor2 by f20f26; factor3 by f30f36; factor1 WITH factor2 @1; factor2 WITH factor3 @1; factor3 WITH factor1 @1; Best regards and thank you in advance. 


The M0 model is just the onefactor model, right? 


Yes, but from the examples in the Webnote describing the strictly positive SB Test i thought in order to compare the models i would need the same notation in the syntax (i.e. 3 correlations set to one instead of just "factor1 by f10f36") so that i can save the start values for the M10 Model and then free the correlations between the factors. 


If you are using the MLR estimator with categorical data you should use the unscaled likelihood ratio test. The SB is designed to be used for the case when you are treating the variables as continuous. Strictly speaking there is a bit of a problem in using LRT for this purpose (overfactoring) see http://www.statmodel.com/download/Schmitt%202011Jour%20of%20Psychoed%20Assmt%20%20EFA%20and%20CFA.pdf and Hayashi, K.., Bentler, P. M., & Yuan, K. –H. (2007). On the likelihood ratio test for the number of factors in exploratory factor analysis. You might want to consider using BIC as well. 


Dear Tihomir, thanks for your detailed reply. I am using MLR because the complex sampling option (type=complex) requires it. Aside from that i am not treating the data (5pointscales) as categorical (yet). Best regards, Malte 


We need to see the relevant files and your license number at support@statmodel.com to help you further. 


Hi I have been working with ESEM models with MLR estimator. To compare nested models I have been using SB x^2 and it has worked fine until I got a negative estimate when comparing two models. I found your web note on how to compute the strictly positive SB x^2 but when I tried to run the M0 model with the svalues command to get the starting values for the M10 model I got the following warning text in Mplus: *** WARNING in MODEL command The SVALUES option in the OUTPUT command is not available with the use of EFA factors (ESEM). Request for SVALUES will be ignored. Do you have any advice for me how to proceed? Best regards Johan 


You can just use the final results in the M0 output and put these values (manually) as starting value for M10 run. The SVALUES option is just a convenience feature that does this for you but you can do it manually as well. The strictly positive SB chisquare is not very easy to do with ESEM. You have to work with the unrotated model. We may eventually update the web note to include a step by step computation for the strictly positive SB for ESEM. Consider using Model Test as a simpler alternative. 


Thank you for your quick response. I think I will use the Model Test command. Have a nice day 

Paula Vagos posted on Friday, December 07, 2012  9:20 am



Hello, I have been trying to compute the SB chisquare difference test, bu my cd is negative, and so I need to create the M10 model. But I dont understand how to do it Can anyone help me? Thanks 


Send the outputs from the m0 and m1 analyses along with your attempt at the m10 model. Include your license number. 

Sabrina posted on Saturday, April 27, 2013  3:04 pm



Hello. I computed the chisquare difference test for 2 nested models and it was negative so I followed the instructions in WebNote 12, but I received the following error message and can't figure out what to do next? THE MODEL ESTIMATION TERMINATED NORMALLY THE STANDARD ERRORS OF THE MODEL PARAMETER ESTIMATES COULD NOT BE COMPUTED. THE MODEL MAY NOT BE IDENTIFIED. CHECK YOUR MODEL. PROBLEM INVOLVING PARAMETER 131. THE CONDITION NUMBER IS 0.494D04. THE ROBUST CHISQUARE COULD NOT BE COMPUTED. 


Please send the output and your license number to support@statmodel.com. 


Hello, I have a longitudinal crosslagged path model with four outcomes over three assessment points. Three of the outcomes are continuous and the fourth is a count, so I am using MLR. I also am using the knownclass method to compare invariance across males and females. My analysis command looks like this: Type = mixture ; Algorithm = integration ; Integration =montecarlo ; I want to test whether two parameters in the model are equal to each other (although they are invariant across sex), so I constrained them to be equal and compared this to a model where they were free. However, I get a negative scaled chisquare difference value. I read webnote #12 and tried to estimate model 10, but when I pasted the start values from model 0 in, the new model does not converge. I'm sure I'm doing something wrong but am not sure what. 


Use MODEL TEST. Label the parameters in the MODEL command. MODEL TEST: 0  p1 p2; 

jtw posted on Friday, September 20, 2013  5:52 am



Hi there, I am comparing the fit of nested models (i.e., bifactor, correlated traits, higherorder, unidimensional) using difftest. I have clustered data and obtain different conclusions when type=COMPLEX is used as compared to when I do not adjust for clustering. Specifically, the bifactor model is preferred when I do not account for clustering, whereas the correlated traits model seems to be preferred when I account for clustering. It may be helpful to know that I obtain all chisquare difference test statistics when not adjusting for clustering. However, I do not obtain the chisquare the difference test statistic for the correlated traits model when adjusting for clustering (because the chisquare value in this case is actually lower than the value for the bifactor model; i.e., the models are not technically nested). Also, it is worth noting that there are no real substantive differences with respect to any of the factor loadings in either case; so, the only issue at hand is which model is preferred. Any recommendations of which results should be reported in this case? Thank you in advance for your time. 


If you have clustered data, you need to do the analysis taking that into account. 

jtw posted on Tuesday, October 08, 2013  9:07 am



Hi there, I understand nested models require two things: 1) more restrictions (higher degrees of freedom) AND 2) worse fit (which can be examined via Tech 5 output). When a model is more restricted yet has slightly better fit, MPlus does not execute the DIFFTEST and returns the error indicating that the models are not nested. In this specific situation, is the conclusion to be drawn: 1) the more restricted model (e.g., correlated traits) is preferred over the less restricted model (e.g., bifactor model); or 2) no firm conclusions should be drawn because the DIFFTEST is not executed. I'm thinking the answer is #1. Am I right? However, if it is #2, what can be done? Thank you in advance for your time. 


You cannot compare the chisquare values of the estimators that end in V, WLSMV and MLMV. You would need to compare their fitting functions which you will find in TECH5 at the bottom of the first column. The lower the fitting function the better. The conditions that you mention are necessary but not sufficient conditions for nesting. Please send the outputs and your license number to support@statmodel.com for further information. 

Back to top 