PPP for Bayesian analysis
Message/Author
 Johnny Zhang posted on Monday, November 14, 2011 - 1:18 pm
I have a question regarding PPP when using the Bayesian estimator. In Muthén, B. & Asparouhov, T. (2011), it says low PPP indicates poor fit but also says a PPP around 0.5 indicates an excellent-fitting model. However, in my analysis, I had a PPP about 0.99. Did this indicate a poor fit model or a good fit model? Thanks.
 Linda K. Muthen posted on Tuesday, November 15, 2011 - 9:31 am
Good fit.
 Rohini Sen posted on Thursday, May 03, 2012 - 7:20 am
If PPP is around 0.36, is that a very poor fit?
 Linda K. Muthen posted on Friday, May 04, 2012 - 10:02 am
This would indicate good fit. You are looking for values greater than .05. A value of .5 would indicate excellent fit.
 Phil Wood posted on Tuesday, November 27, 2012 - 8:35 am
Is it ever acceptable to use the ppp from a Bayesian analysis to compute a Bayes Factor between two models, or is it always preferable to use the BIC? (Assuming you have enough draws to believe it as a point estimator). I read a recent article by Meng in the Annals of Statistics which frowned on doing so, but it seems that argument is based on "using the same data twice," which we also did when using the BIC. Any thoughts from anyone?
 Tihomir Asparouhov posted on Monday, December 03, 2012 - 9:54 am
Phil

I haven't seen any methodology on computing BF using PPP. PPP definition typically involves just 1 model while BF involves 2 models. Note also that PPP can be defined in many ways and the way it is defined in Mplus has nothing to do with what is in Meng's article. Mplus uses SEM style chi2 fit function.

If you want to compare PPP and BF testing power and quality I would recommend looking at a simulation study.

Tihomir
 Phil Wood posted on Tuesday, December 04, 2012 - 7:25 am
I had just meant dividing the PPP from one model by the PPP from another model. Looking at just a few calculations, it doesn't sem to work very well in practice relative to using, say, the BIC. Thanks for clearing up my confusion on Meng's article, though!
 Yoonjeong Kang posted on Thursday, April 03, 2014 - 9:30 am
Hello.

I am running a second-order factor model using both ML estimation and Bayesian estimation. I have 16 subscale scores and 16 subscale scores load on four factors and the four factors load on one higher factor.

I found that Model fits from ML estimation seem to be good based on CFI(0.973) and RMSEA (0.044). However PPP from Bayesian estimation was very low. It was PPP=0.00. I don't understand why two estimation methods provide very different results in terms of model fit. Do you have any idea why this happens?
(For the reference, I used default priors in bayesian estimation.)
 Bengt O. Muthen posted on Thursday, April 03, 2014 - 9:47 am
PPP is more related to chi-square than to CFI. What is your sample size, chi-square value, df, and p-value?
 Yoonjeong Kang posted on Tuesday, April 08, 2014 - 11:21 am
Dear Dr. Muthen,

Sorry for the late response. I have a sample size of 2,200. I got a chi-square value of 689.058, df of 101, and p value of 0.000 using ML.

 Bengt O. Muthen posted on Wednesday, April 09, 2014 - 10:57 am
So the p-value for the chi-square and the PPP agree as is expected. These fit statistics are less forgiving than CFI. So it isn't a matter of Bayes vs ML but a matter of which fit statistics you deem best.
 Yoonjeong Kang posted on Wednesday, April 09, 2014 - 1:59 pm
Dear Dr. Muthen,

I prefer using other fit indices such as CFI and RMSEA than the chi-square when assessing the model fit. It's because the literature has demonstrated that the chi-square is sensitive to sample size.

I wonder whether PPP is also sensitive to sample size or not. If so, I wouldn't want to use PPP to assess model fit. Then my another question is that is there any other model fit indices to assess model fit in Bayesian SEM?

 Bengt O. Muthen posted on Wednesday, April 09, 2014 - 2:13 pm
I don't think one should be so quick to abandon chi-square (or PPP). Instead, try to investigate why the chi-square is not good enough. What's good about the Bayes approach is that you get an interval for the fit and when you relax the model you can see how the lower limit decreases, getting closer to zero (and into the negative).

PPP is based on chi-square and therefore also has its power increase with increasing sample size. Mplus does not give any other Bayes fit measures. Note, again however that I would think you want to relax some restrictions in your model as discussed in

Muthén, B. & Asparouhov, T. (2012). Bayesian SEM: A more flexible representation of substantive theory. Psychological Methods, 17, 313-335.

If you do so, some of your key results might change.
 anonymous Z posted on Tuesday, February 24, 2015 - 1:56 pm
Hi Dr. Muthen,

I just started to use Bayes estimation, and I have two questions.

1. The output showed

"THE MODEL ESTIMATION TERMINATED NORMALLY

USE THE FBITERATIONS OPTION TO INCREASE THE NUMBER OF ITERATIONS BY A FACTOR OF AT LEAST TWO TO CHECK CONVERGENCE AND THAT THE PSR VALUE DOES NOT INCREASE."

Is this a warning/error message?

2. model fit: I got a model fit as below.I assume it is good since p=0.515. But I don't understand the confidence interval info. Is it good or bad?

95% Confidence Interval for the Difference Between the Observed and the Replicated Chi-Square Values

-18.234 17.604

Posterior Predictive P-Value 0.515

Thank you very much!
 Bengt O. Muthen posted on Wednesday, February 25, 2015 - 1:07 pm
1. No, it's a suggestion for good analysis practice.

2. Read my into Bayes paper on our website.
 Jana Holtmann posted on Wednesday, December 02, 2015 - 3:20 am
Hi,

I've got a question concerning the interpretation of the output on PPP values in Monte Carlo Simulations using Bayesian estimation. The information given on the PPP value is a table with expected and observed proportions, however no percentiles. I understand that the PPP does not follow a known distribution, hence no percentiles? I was wondering how to interpret the expected proportions then. Is it correct if I assume that the expected proportions listed for the PPP are just the possible values of the PPP (and the observed column then gives me the cumulative frequency of values observed in the respective range) ?

Thanks!
 Linda K. Muthen posted on Wednesday, December 02, 2015 - 6:54 am
The description of how to interpret that output is given on page 412 of the current user's guide. This is in Chapter 12 under the title Monte Carol Output.
 Jana Holtmann posted on Wednesday, December 02, 2015 - 7:20 am
Unfortunately the info in the user guide doesn't exactly answer my question. On page 412 it says that "The column labeled Proportions Expected (column 1) should be understood in conjunction with the column labeled Percentiles Expected (column 3)."
But there is no column with Percentiles in case of PPP values.

Thanks!
 Linda K. Muthen posted on Wednesday, December 02, 2015 - 10:15 am
 Freya Glendinning posted on Tuesday, June 07, 2016 - 5:24 am
Dear Linda and Bengt,

Please can you help me understand why I get two identical PPp-values and 95% C.I's in two separate models where the IV's are the same but the DV is different?

There are no latent variables in either model and each model has the same number of parameters.

Freya
 Bengt O. Muthen posted on Tuesday, June 07, 2016 - 8:53 am
 Freya Glendinning posted on Tuesday, June 07, 2016 - 8:58 am
Hi Bengt,

I am using the universities student license which doesn't cover support.

Thank you,
Freya
 Bengt O. Muthen posted on Wednesday, June 08, 2016 - 10:07 am
This can happen if your DV versions are linear transformations of each other. Beyond that, we can only tell from the output and data.
 Nate Breznau posted on Friday, October 07, 2016 - 2:31 am
Can you provide references for the PPP as indicating good fit at 0.5? Also, references to better understand and interpret convergence of chi-square with the Bayes Predictive Probability chi-square?
 Nate Breznau posted on Friday, October 07, 2016 - 3:21 am
I found a brief discussion of PPP-values in:

Muthén, Bengt and Tihomir Asparouhov. 2012. “Bayesian Structural Equation Modeling: A More Flexible Representation of Substantive Theory.” Psychological Methods 17(3):313–35.

However, more references, especially to the latter question, would be appreciated!
 Bengt O. Muthen posted on Friday, October 07, 2016 - 9:46 am
The best ref is the 2014 Gelman et al Bayesian Data Analysis book, 3rd ed.
 Freya Glendinning posted on Thursday, February 02, 2017 - 3:49 am
Dear Muthen and Muthen,

I have a 3 factor bayes cfa that is producing a PPP of .7 95% C.I = -87, 32
My interpretation is that a PPP of .7 would indicate good fit and would be equivalent to a PPP of .3 (if say the C.I was reversed to -32, 87) Would you agree?

Freya
 Tihomir Asparouhov posted on Friday, February 03, 2017 - 8:53 am
Freya I would not quite agree with reversing the confidence limits. Either way the PPP indicates a good model fit and there is no evidence to reject the model.
 Freya Glendinning posted on Friday, February 03, 2017 - 9:01 am
Great, Thanks Tihomir.
 Freya Glendinning posted on Monday, February 06, 2017 - 1:41 am
Sorry Tihomir I just want to follow that up with another question because I think my confidence limit reversal example was a bad way to frame it...

I have run models that have had a PPP of .5 where the confidence limits centre around zero. When I have had PPP's of approx. .1 the upper limits (e.g. 90) have usually been further from zero than the lower limit (e.g. -10) and when I have had PPP's of approx. .9 the lower limit has been much further from zero (e.g. -120) than the upper limit (e.g. 20)...

So, if a PPP of >.05 indicates good fit and .5 excellent fit would it be correct to think that when the PPP increases past .5 fit starts to worsen (as reflected in the confidence limits)? If so does this mean a PPP >.05 and <.95 indicate good fit?

Thanks again,
Freya
 Tihomir Asparouhov posted on Monday, February 06, 2017 - 8:04 pm
I would disagree. PPP should be interpreted similarly to how P-value is interpreted. If the SEM test of model fit P-value is >0.95 it is an unusually good fit and most likely a more restricted model will also fit well but I would not call this a poor fit.

Note this is very specific to this PPP value that Mplus computes for SEM models using this particular discrepancy function which is the regular SEM chi-square.

Almost all other PPP values used in the statistical literature that are based on different discrepancy function obey the logic that you describe - but not this one.

Also consider this: asymptotically - with large samples and normal data the PPP value is almost identical to the p-value.
 Freya Glendinning posted on Tuesday, February 07, 2017 - 3:07 am
This now makes a lot of sense... I have imposed stricter priors on both cross-loadings and residual covariance's and as you said, the model still fits very well.

Thanks for this Tihomir!
Freya
 Chi Hang Au posted on Sunday, August 06, 2017 - 1:27 pm
Hello,

Is it possible to save out the posterior-predictive distribution used to compute a ppp value as a separate file?

Thank you!
Chi
 Bengt O. Muthen posted on Sunday, August 06, 2017 - 4:44 pm
No.
 Maria Pavlova posted on Tuesday, December 05, 2017 - 2:13 am
Hi,

I am estimating highly complex twolevel models with multiple mediators, some of which are categorical (binary and ordinal) variables. All mediation occurs at the first level. I use Bayesian estimation. I first estimated all paths (first and second stage of mediation) in separate models, which converged and had excellent to acceptable fit. When I estimated the full mediation model, it converged but had a terrible fit, and estimated effects deviated from those from the separate models. I included all covariances between the predictors at each level and all covariances between the mediators, so I cannot see the source of misfit. Model parameters: 444 free parameters, 95% Confidence Interval for the Difference Between the Observed and the Replicated Chi-Square Values 4137.377 4518.237, ppp < .001.
I have also tried to estimate the second stage of mediation separately in order to use an online mediation calculator that multiplies the paths and uses bootstrap. To get categorical mediators to be treated as latent variables as at the first stage, I specified them as categorical variables. (However, they are actually predictors in this model as only one equation - second stage - is being estimated.) These models don't converge, no matter what I try, not even a simple covariance matrix.
I am just wondering whether I miss some essential tip on how to estimate such models. Thanks in advance!
 Bengt O. Muthen posted on Tuesday, December 05, 2017 - 12:25 pm
We need to see the separate and full model outputs - send to Support along with your license number.
 Hazel B posted on Friday, April 20, 2018 - 4:12 pm
Dr. Muthen,

I read some of your articles, but I am still wondering if the model fit can be determined based on the number of free parameters and 95% confidence interval. (Like when we use ML estimator, we sometimes conclude that the model fits the data well based on other values (e.g., CFI, TLI...), even if the p-value is less than .001.)

Number of free parameters: 119
(119 * 2 = 338)
95% confidence interval: 35.028 - 144.421
PPP: 0.001

 Hazel B posted on Saturday, April 21, 2018 - 1:29 pm
Nevermind. I understand that the positive lower limit indicates a poor fit.
 Tihomir Asparouhov posted on Saturday, April 21, 2018 - 3:46 pm
That is correct. If you are interested in establishing concepts such as "approximate fit" (which is what CFI, TLI do) you should look into BSEM and these three articles