Hicham Raïq posted on Monday, January 21, 2013 - 9:50 am
I read the article of Kristopher J Preacher "The Problem of Model Selection Uncertainty in SEM". I want to know if is it mandatory to quantify the variability in selection criteria? The author suggests to use the boostrap and I don't know if it will be adapated to my analysis because I use a compelex data with stratification option. I m asking this question because the BIC is my only goodness fit criterion as I m woking with categorical varaibles. And I want to make sure that this cirterion is robust.
It is not mandatory, but our study suggests that sampling variability in selection criteria like BIC is higher than what many assume, and not something that can be ignored forever. We were unable to suggest a foolproof way to quantify the sampling variability in BIC-based model selections on the basis of a single sample.
However, we did tentatively suggest that researchers consider nonoverlapping CI's for the expected BIC (EBIC) as conservative evidence for the distinguishability of pairs of models. Obtaining a CI for BIC is not very easy, but can be done with any programming language that includes a noncentral chi-square distribution function, like R.