answer1-answer5 with answer6; answer1-answer3 with answer5; answer1-answer2 with answer3; answer1 with answer2;
I generated 5 multiple impuations using Bayesian estimation with plausible factor scores saved. Using 5 imputed data sets I ran a multilevel CFA using WSLMV repeating the same model that was used for the imputation.
I want to examine how these plausible factor scores correspond to the behaviour of the True scores from my factor model.
Where I am confused is in how to go about reproducing the within-level factor correlations from the CFA model using my plausible factor scores. I would like to see how closely the plausible factor score correlations correspond to the correlations between my latents in my within-level CFA.
Any adivce on how to proceed would be very welcome?
It seems like there are at least 3 ways to find the correlation between the 2 factors. (1) The 2-level Bayes run that you show gives a Bayes factor correlation estimate. (2) The WLSMV run based on the imputed data gives a WLSMV factor correlation estimate. (3) And the plausible values that your Bayes run generates can be used to compute the factor correlation.
As for (3) you get N*n factor scores for each factor, where N is the number of plausible draws and n is your samle size. You get the factor correlation from those N*n factor scores.
Also, take a look at this paper on our website:
Asparouhov, T. & Muthén, B. (2010). Plausible values for latent variables using Mplus. Technical Report.
I think I didnt explain myself clearly. I was interested in examining how the plausible factor score estimates correlated with their true scores, that is, the within-level factors from my CFA. Is there any reason why I could not enter the plausible factor scores into my multilevel CFA as covariates (as below) in order to see how well they correlate with their respective true factor scores?
F1 BY answer1* answer2 answer3; F2 BY answer4* answer5 answer6;
F1 WITH F2; F1 WITH PlausibleF1; (Correlation of Plausible factor socre with true factor score) F2 WITH PlausibleF2;(Correlation of Plausible factor socre with true factor score) PlausibleF1 WITH PlausibleF2;(correlation of plausible factor scores)
%between% answer1-answer5 with answer6; answer1-answer3 with answer5; answer1-answer2 with answer3; answer1 with answer2; PlausibleF1 WITH answer1 to answer6; PlausibleF2 WITH answer1 to answer6;
Any adivce on how to proceed would be very welcome?
I am not sure that a high correlation from that approach is indicative of high-quality plausible values. I would instead look at how the plausible values for the two factors relate to each other and to other variables in comparison to estimating those quantities directly in the model.
If you are doing a Monte Carlo study you could you see how true scores (generated factor values) compare to plausible values.
I agree that looking at how the plausible values for the two factors relate to other variables in comparison to estimating those quantities directly in the model is useful.
Inspite of having a good fitting model I need to provide day-to-day users of the instrument (eg,clinicians) with confidence that a scoring method has correlational accuracy, univocality and that the scoring method (factor score estimates or plausible values) correlate with their true scores. This is because even for highly determinate factors (which mine are) I could still end up chosing a poor set of factor score estimates - these need to be evaluated somehow. Factor score estimates - Nunnally:
“If the multiple correlation [the proportion of determinacy in the factor] is less than .70, one is in trouble. In that instance the error variance in estimating the factor would be approximately the same as the valid variance. At a very minimum, one should be quite suspicious of factor estimates obtained with a multiple correlation of less than .50, because in that case less than 25 percent of the variance of factor scores can be predicted from the variables. Then one could not trust the variables as actually representing the factor.” (1978, p. 426).
Is it that the method that I suggested is technically incorrect to answer the question, or is it that you think it is not a useful test or not as useful as the other method you proposed?
von Davier M., Gonzalez E. & Mislevy R. (2009) What are plausible values and why are they useful? IERI Monograph Series Issues and Methodologies in Large-Scale Assessments. IER Institute. Educational Testing Service.
Both actually but for the moment, the former, the quality of scores in terms of individual's scores. I tried using regression weighting methods using the factor score coefficient matrix generated from a within-level polychoric correlation matrix with ML estimation and several unit weighted and course factor score methods but these all seem to perform very poorly using the method I pasted as syntax. The factor score estimates all correlate with their true scores no higher and often a lot lower than r=.65. (that's r not r-square).
I take it by your question, plausible values may not be suitable for individual use? I suppose this would make sense if we're using imputations as these are designed to estimate population parameters and standard errors and not individual's responses.
I will continue to read on the subject of plausible values but I'm still stuck with the problem of having a good approximate fit but no method for users to score the instrument with confidence. few people ever seem to bother evaluating their factor scoring methods for instruments so I wanted to make an extra effort to learn about it.
I think plausible values have an advantage over regular estimated factor scores in that each individual gets a distribution of values so that the uncertainty is clearly presented and the shape of the distribution is clear (an estimated factor score for an individual at most gets a SE). This is useful to have when you compute the variance over people and when you compute the relationship to other variables (see the section 4.2 of our imputation paper that I referred to). I don't know if I have seen that the average plausible value for a certain individual is any better than the estimated factor score for that individual.
Factor scores (and plausible values) have different strengths and weaknesses for different uses as was discussed already by Tucker in the 50's. For example, ranking individuals is one use, regression another. For a recent discussion, see Skrondal, A. and Laake, P. (2001). Regression among factor scores. Psychometrika 66, 563-575.
I think the evaluation approach you suggest may have a variation on the Heisenberger (?) problem - in evaluating the plausible values you alter the factor meaning. The factors become determined also by the
F WITH Plausible;
statements, which is not what you want.
Again, to really see how well the factor scores or plausible values work in the model and intended use situation, you want to do a Monte Carlo study so that you can compare the true, generated scores with your estimated ones.