Composite reliability for categorical... PreviousNext
Mplus Discussion > Categorical Data Modeling >
Message/Author
 claudio barbaranelli posted on Saturday, March 22, 2014 - 7:25 am
Hi,
In 1981 Fornell & Larcker presented coefficients for estimating the reliability of indicators and of constructs from the results of a CFA (using "lambdas" and "thetas").
In their 2011 book Raykov and Marcoulides introduced (formula 7.17, page 161) the composite reliability which has the same formula of Fornell and Larker "reliability for construct" acknowledging its similarity with omega (McDonald, 1999).
Raykov and Marcoulides warn students to use these coefficients when items are dichotomous or when they have 3 or 4 possible ordinal responses.
Let me now turn to my question:
since in these formulas only parameters from factor solution are used (the lambdas and the thetas, and also the phis in case of a multidimensional structure) what assumptions are violated if these parameters come from a solution derived from the analysis of polychoric or tetrachoric correlations, like for example from an Mplus solution using WLSMV estimators ?
Moreover, FACTOR software (Lorenzo-Seva & Ferrando, 2013) gives estimates for omega as well as alpha and other reliability coefficients also when a tetrachoric or a polychoric correlation matrix is analyzed from raw data (ULS estimator is recommended in this case). I assume that these estimates can be trusted, but this would be at odds with what suggested by Raykov and Marcoulides.
So ?

Claudio
 Linda K. Muthen posted on Saturday, March 22, 2014 - 11:48 am
Where in the book do Rakov and Marcoulides warn against using these coefficients with categorical items?
 claudio barbaranelli posted on Sunday, March 23, 2014 - 1:10 am
Linda
is their book "Introduction to Psychometric theory" (2011). It is at page 176, paragraph 7.7. Reliability evaluation with categorical items

"The preceding methods of reliability evaluation discussed in this chapter are not really applicable with items of this kind"

Items of this kind are "dichotomous(binary)..." and items with "only three or four possible ordinal responses"

The "preceding methods" are composite reliability (omega) and all the other variants discussed in their chapter 7, methods the use parameter estimates of CFA (lambdas, thetas, phis).

Best
Claudio
 Bengt O. Muthen posted on Sunday, March 23, 2014 - 12:52 pm
I don't have a succinct answer to give you. I think basing reliability estimates on latent correlations such as tetrachorics gives some information, but this focuses on the continuous latent response variables underlying the categorical observed variables, so you don't get a reliability of the observed sum of items. Tetrachoric-based reliability was actually Linda's 1983 dissertation topic. My 1977 dissertation had a section on the reliability of the observed sum of binary items as a function of the quality and number of items. In my view, reliability for say binary items has been overtaken by the IRT focus on test information curves for the factor.
 claudio barbaranelli posted on Sunday, March 23, 2014 - 2:51 pm
Thank you Bengt. Let me just complete the issue.
I have items with 4 ordered categories.
If I base my reliability estimate on MLM then these estimate are biased (maybe downard ?).
If I base reliability estimates on WLSMV actually I am not assessing the reliability of observed items but of their latent underying var (and this will a kind of upward bias for the observed ?).
What to do ? Report both ?
Was Linda's dissertation published ?
Thanks
Claudio
 Bengt O. Muthen posted on Monday, March 24, 2014 - 8:26 am
You are probably right about downward and upward bias. But, with 4 ordered categories, treating them as continuous may not be a bad approximation if there are no strong floor or ceiling effects.

But I often wonder - why the interest in reliability? Is that because the intention is to sum up the items? In which case they are treated as continuous. If you really have a 1-factor model in mind, why not focus on how well the factor is measured - which then leads back to IRT's information curves.

Linda's dissertation was not published, although later work by Marcoulides took this approach to G-theory. Nor was my contribution published.
 claudio barbaranelli posted on Monday, March 24, 2014 - 11:37 am
Thank you so much Bengt.
Actually I am working on a scale developed by other colleague, that's quite messy...
This scale has 3 subscales.
The first two are multidimensional
the last unidimensional.
However, in previous papers, although the
multidimensionality, Cronbach's alpha has
been always used to assess scale reliability also for the 2 multidimensional ones.
This is why I was interested in the measures of composite reliability proposed by Raykov and Marcoulides in their 2011 book. But there I found the warning on using these indices on items with less than 5 ordered categories...
Well actually the items have some excessive skewness, but there are no ceiling or floor effects, so I performed MLM to analyze them wiht CFA and used results from this CFA for computing composite reliability for multidimensional structures. I did the same using WLSMV and of course estimates were much higher.... So this is the story... My feeling is to present both estimates ....
Anyway, thank you so much again to you and to Linda.
Mplus is number 1 in many many things: one of them is customer orientation.
Best
Claudio
Back to top
Add Your Message Here
Post:
Username: Posting Information:
This is a private posting area. Only registered users and moderators may post messages here.
Password:
Options: Enable HTML code in message
Automatically activate URLs in message
Action: