where Li= the standardized factor loadings for the factor Var(Ei)=the error variance associated with the individual indicator variables.
bmuthen posted on Thursday, February 20, 2003 - 7:42 am
I am not familiar with the formula for composite reliability, but it would seem that the summation of item loadings without weighting them would require that they are standardized, and probably using StdYX.
Anonymous posted on Tuesday, May 11, 2004 - 7:41 am
Prof.Muthen, I wanna compute 4 factoes' reliability following your method(Multilevel CSA, 1994), may I and how I get each level 2 and level 1 factors variance and their related error variacen by Mplus 3.0 output? for level 2 factore reliability, is the formula above the right one? sorry I can
If you are asking how to obtain model estimated variances for your factors, ask for TECH4 in the OUTPUT command.
Anonymous posted on Tuesday, May 11, 2004 - 11:41 pm
does TECH4 can also give the error variance related on specific factor? or a I have to compute the total/error variance for each factor? thanks for your guide! by the way, how can I register this disscussion list?
I am interested in computing the composite reliability of a number of survey items that I have analyzed in a CFA model fitted via Mplus.
In a series of SEM Journal articles, Raykov demonstrated that the composite reliability for a set of continuous items is explainable as:
var(true) / [var(true) + var(residual)]
within a CFA model where var(true) is equal to the sum across items of the variance explainable for each item and var(residual) is the sum of the residual error variances.
This is essentially an intraclass correlation coefficient. Of note, if the CFA model postulates a single factor with equal loadings and equal residual values, the value of this expression is identical to Cronbach's coefficient alpha. When loading values are unequal, this measure will yield more accurate reliability estimates than alpha (alpha will be underestimated).
In my present situation, however, I am analyzing ordered categorical outcomes. It is possible using MODEL CONSTRAINT in Mplus to compute var(true) and var(residual) for ordered categorical outcomes, just as one can do for continuous outcomes.
I asked a colleague at AERA for his opinion of the usefulness of this practice, and he expressed doubt that the resulting statistic would be in interpretable as a measure of composite reliability of the observed items. He thought it would instead be an index of reliability of the underlying latent y* values rather than the y values themselves. I am curious about this conclusion, though, given that direct modeling of the y's is mathematically identical to modeling of the y* values, per the discussion in the Mplus user's guide technical appendix.
I'm just getting started researching this issue in the literature, but did come across the following article:
Roberts, Chris & McNamee, Roseanne. Assessing the reliability of ordered categorical scales using kappa-type statistics. Statistical Methods in Medical Research, 2005, 14: 493-514.
On page 498, they present what they refer to as an intraclass kappa coefficient for ordered categorical data and express it as the ratio of between-subject to total variance.
Do you have any thoughts/opinions you can share about the usefulness of the approach I proposed above (computing the explained variance / [explained variance + residual variance] in Mplus) and the appropriateness of interpreting the resulting coefficient as a reliability estimate of the composite of the y values?
Big topic. In factor analysis with categorical outcomes, the logit or probit regression of an item on the factor(s) is a regular logistic or probit regression and therefore can be expressed in terms of a y* dependent variable that has a linear regression on the factor(s) and that is crudely observed by a categorical y. While those 2 formulations are identical for regression relations, reliability is another matter. The y* variables have residual variances, although they are not identifiable parameters, but instead remainders adding up to a y* variance of one. So, one can formulate reliability in terms of y* and this was done by Linda Muthen in her 1983 dissertation. Others have picked up that idea too; I think Marcoulides wrote something on it later on. This type of reliability is indirectly relevant for the observed items. But you have to ask what kind of aggregation of the items you are interested in getting the reliability for. Is it for the sum? Or shouldn't you simply ask - what is the reliability of the factor scores that you get? The answer to that question is in the IRT literature and points to information curves, the graphics of which is something that will be included in the soon forthcoming Mplus version 4.1
Thank you, Bengt. This is very helpful. Do you have any favorite references that discuss the use/interpretation of information curves as they will be implemented in Mplus 4.1? It would be helpful to read up on these in advance of the release of v4.1 of Mplus so that I can hit the ground running when 4.1 becomes available.
MODEL: r1 BY y1@1; r2 BY y2@1; r3 BY y3@1; r4 BY y4@1; r5 BY y5@1; r6 BY y6@1; y1-y6@0; f1 BY r1* (l1) r2-r4 (l2-l4); f2 BY r3* (m1) r4-r6 (m2-m4); f1-f2@1; f3 BY; f3@0; f3 ON r1-r6@1; f4 BY; f4@0; f4 ON f1 (o1) f2 (o2); f3 WITH f4@0; Model constraint: o1=l1+l2+l3+l4; o2=m1+m2+m3+m4; Output: TECH4;
Where the first factor loads on items y1-y4, and the second factor loads on items y3-y6. The reliability of the sum score of the observed variables is estimated by the quotient between the estimate of the true composite variance (F4) and the variance of the composite (F3), both reported in TECH4.
Now, I want to use this model with ordinal data (and the WLMV estimator), and a very similar noncongeneric scale with 6 items and two factors. My question is: what adjustmensts do I need to make?, is it possible to run this model under the delta parameterization?, and if it is necesary to use the theta parameterization, what other adjustmensts do I need to make?
Without me getting into the tech doc you refer to, here are some reactions, first regarding the setup you show for continuous outcomes and then for categorical outcomes.
Do you really need to complicate the input by defining the factors r1-r6 behind each observed outcome? Why not have f1 and f2 measured by y1-y6 directly?
I don't think statements of the kind
work. You can define the factor by any observed variable and not adversely impact the model by fixing the loading @0.
When switching to categorical outcomes, the question is if you want to work with ML or WLSMV estimation - which is also a choice between a logit and a probit model. I guess in this situation it makes a difference if you put a factor behind each of the outcomes because the regression of f3 on r1-r6 is a regression on continuous latent response variables, whereas the regression of f3 on y1-y6 is a regression on categorical outcomes with ML (continuous latent response vbles with WLSMV). Which to choose is a research question that I don't get into here. With WLSMV, I don't see that the Delta/Theta choice makes a difference.
Thank you very much for your quick and enriching answer and … Wonderful!, I did a direct translation from Lisrel to Mplus (and it worked), but I forgot that Mplus is more flexible, so in Mplus the r1-r6 aren’t needed. The code reduces to:
MODEL: f1 BY y1* (l1) y2-y4 (l2-l4); f2 BY y3* (m1) y4-y6 (m2-m4); f1-f2@1; f3 BY; f3@0; f3 ON y1-y6@1; f4 BY; f4@0; f4 ON f1 (o1) f2 (o2); f3 WITH f4@0; Model constraint: o1=l1+l2+l3+l4; o2=m1+m2+m3+m4; Output: Standardized; TECH4;
And it gives the same reliability results as Raykov’s.
The statements: f3 BY; and f4 BY;, do their work perfectly (and they give the same results that, i.e. f3 BY y1@0).
When trying this model with ordinal data (y1-y6), and the WLSMV estimator, I receive the message: “The model is not supported by DELTA parameterization. Use THETA parameterization.”. I have done it, and then the model works. Nevertheless, when using the ML estimator you get an error, “Internal Error Code: PR1004 - Parameter restriction split problem”.
Thanks for exploring - I learned something new. I see now that Theta parameterization won't work, because the model falls into the category described in the User's Guide as "categorical dependent variable is both influenced by and influences either another observed dependent variable or a latent variable." The ML issue is one of Model constraint implementation where currently parameters of certain different types cannot be part of the same constraint.
When using Mplus, AVE is calculated by taking the standardized estimated loadings for each item within its respective construct. The value of the estimate is squared, and then summed to create the numerator of the AVE statistic. The same value for the estimate is used in the denominator, only this time the estimate and 1 minus the estimate are used. You acn refer to Gefen, Straub and Boudreau (2000) who use the symbol lowercase lambda, l, to represent the value of the estimate. I'll use "E" to represent sigma. The formula for generating the AVE statistic is then AVE = (Eli^2)/((Eli^2)+(E1-li^2)) You then take square root of the resulting statistic and places it within the correlation table of the latent constructs, which is generated using the “tech4” option in the Mplus output command. That value is compared to the correlations of that construct to the other constructs in the model. If the square root of the AVE for a construct is above .50 and larger than its correlation with other constructs, convergent and discriminant validity are said to be shown (Gefen et al. 2000). Hope this helps!
Dear all, I) With the help of your posts, I transfered Raykov's scale reliability into this command: ANALYSIS: parameterization=theta; MODEL: F1 BY item1* (l1) item2-item5 (l2-l5); item1-item5 (ve1-ve5); F1@1; MODEL CONSTRAINT: NEW (RELIABF1); RELIABF1 = (l1 +l2 +l3 + l4 + l5)**2/ ((l1 +l2 +l3 +l4 +15)**2 + (ve1 +ve2 +ve3 +ve4 +ve5)); OUTPUT: Standardized tech4;
However, I get an error message: THE MODEL ESTIMATION TERMINATED NORMALLY. THE STANDARD ERRORS OF THE MODEL PARAMETER ESTIMATES COULD NOT BE COMPUTED. THE MODEL MAY NOT BE IDENTIFIED. CHECK YOUR MODEL. PROBLEM INVOLVING PARAMETER 25.
Parameter 25 refers to THETA ITEM1 ITEM1 25
Still, values for estimates, residual variances and reliabF1 are given. If I calculate reliab. by hand (squared sum of unstand. estimate/(squared sum of unstand. estimate+sum of residual variance) result is .80 instead of given .040 (?!)
II) I wasn't sure whether the first line in model command needs to be F1 BY item1* (l1) or simply F1 BY item1. If I do so, I get same error message but lower reliab. (.76) calculated by hand.
I forgot to say, that parameter 25 is theta of the first item. It doesn't matter whether I conduct the analysis separately for my three subscales or altogether. It is always the same error message that occurs, and the corresponding parameter is always the first item.
in which case those residual variances are not identified. You should remove the line:
I also assume that you are using the WLSMV estimator in which case these residual variances are deduced (not estimated as free parameters) from the model and are printed when requesting the Standardized solution in the Output command. So your Model Constraint section will have to use those printed values instead of ve1-ve5.
Actually, with the THETA parametrization, the R-square section seems to be printed only if STANDARDIZED is requested in OUTPUT (I am using Mplus version 6.1).
Anyway, I managed computing Raykov's composite reliability formula by starting from Tina's syntax and applying Bengt's advice. Since the line item1-item5 (ve1-ve5); is removed, the THETA parametrization is no longer needed.
The issue remains however somewhat unclear to me, of how we may interpret Raykov's reliability in the case of ordinal items? In my understanding, it somehow measures the part of true variance in the underlying responses y*_i. How to we relate that to the internal consistency of the actual ordinal responses y_i? I have asked the question on SEMNET, and am waiting for an answer there.
I am afraid I don't have a direct answer to the question of composite reliability with highly discrete items unless they're all binary (in which case our paper with Tihomir and Dimitrov, SEM, 2010, outlines a method of point and interval estimation as well as of the change in it due to revision; see also below in this message).
A procedure for approximate point and interval estimation of reliability of the sum score of discrete items (with up to say 5 levels/values possible) is outlined in our recent book with Marcoulides, Intro to Psychometric theory, 2011, NY: Taylor & Francis. That procedure is not exact and has a potential limitation of not giving a single estimate for the scale's reliability (simple sum score's reliability), as discussed there. With 5 or more levels/values on each item, there's a better procedure outlined in the book for point and interval estimation of the scale's reliability, which uses ML robust.
Many thanks for your prompt response to an actually rather complex issue!
Do you authorize me to forward your response to SEMNET in order to complement the discussion I started there?
Tenko, having browsed your 2011 book with Marcoulides on Amazon, it seems to fit very well my current needs indeed. It should provide me a solid reference to understand fundamental issues in psychometric studies and models, esp. from the latent variable viewpoint. And a detailed treatment of advanced topics such as reliability of ordinal items. I am definitely ordering it!
I think works such as your book should help towards a more widespread adoption of better suited alternatives to Cronbach alpha. Still, given the familiarity of most non-statistician researchers with alpha, I am wondering whether using alternatives will be readily understood when submitting research in non-methodologically oriented journals?
I am new to Mplus but I want to calculate the confidence interval for AVE and CR.
There is a paper: A Comparison of Three Confidence Intervals of Composite Reliability of A Unidimensional Test (YE Bao-Juan 2011) that mentions in the abstract (English) that "...results could be directly obtained by using SEM software Mplus that automatically calculates the confidence interval with Delta method and presents the confidence interval." However, the paper itself is written in Chinese, which I am unfamiliar with.
How do I calculate these statistics with their corresponding confidence intervals?
I think you have a residual variance in your model but want to use the variance in MODEL CONSTRAINT. If that is so, you must define the variance as a new parameter using the components from your model.
Stephen Teo posted on Thursday, March 14, 2013 - 9:28 pm
I am new trying to calculate composite reliability and average variance extracted.
The ways to estimate them are:
Composite Reliability = (Sum of standardized loadings)**2/ ((Sum of standardized loading) **2 + Sum of indicator’s residual variance))
Average Variance Extracted = Sum of squared standardized loadings/ (Sum of squared standardized loadings + Sum of indicator’s residual variance)
My questions are: 1) If the ways estimating Composite Reliability and Average Variance Extracted have anything incorrect, please let me know.
2) Should I use ‘standardized’ residual variances of the indicators in the processes? Or should I use ‘raw’ residual variances of the indicators in the processes?
3) I set seven indicators as categorical items in the CFA analysis (the default estimator for categorical data analysis is WLSMV in Mplus). Mplus outputs showed no residual variance for these categorical items. Is there a way to obtain the residual variances for these categorical items. Or should I use the other ways to estimate composite reliability and average variance extracted for these categorical items?
1) That looks correct. The reliability formula assumes that the factor variance is set to 1. You can also check these general issues which are not Mplus specific on SEMNET. See also the Raykov-Marcoulides book "Introduction to Psychometric Theory".
2) Use raw estimates
3) I don't this reliability formula is for categorical items. Again, ask on SEMNET.
Sorry to bother you again for the same question!! I discuss with my friend (who also using Mplus) about selecting “standardized” or “raw” estimates in calculating composite reliability and average variance extracted. My friend suggested that we should use all raw estimates or all standardized estimates for both factor loadings and indicator’s residual variance in the two processes. Is my friend right? Or should I use “standardized” estimates in factor loadings, but use “raw” estimates in indicator’s residual variance?
Thank you very much for your rapid interpretation!!
I am deeply appreciated for your teaching!!
Tyler Moore posted on Thursday, September 04, 2014 - 9:36 am
I'm estimating a straightforward bifactor model with continuous indicators, and was wondering if the latest version of Mplus can output coefficient omega. Also, I heard it is fairly simple to get confidence intervals for omega using Mplus output, but am having trouble finding specific instructions for that. How do I get those CIs?
I'm attempting to request the composite reliability for factors in a CFA. Each factor has 5 categorical items (0, 1, 2). I used the input above kindly provided by Tina Freyburg but get results that differ from my hand calculations according to the Raykov formula. Following Dr. Muthen's suggestion above, I have used the values from my CFA output under the R-SQUARE Residual Variance heading rather than the ve1 etc. in computing the values.
You don't mention what "why this might be happening" refers to.
Note that the formula you use here is for continuous items, not for categorical items. The latter case is more complex and studied in
Structural Equation Modeling, 17:265–279, 2010
Evaluation of Scale Reliability With Binary Measures Using Latent Variable Modeling Tenko Raykov Michigan State University Dimiter M. Dimitrov George Mason University Tihomir Asparouhov
Cheng posted on Saturday, April 04, 2015 - 11:36 pm
Hi Muthen, I have 3 different measures here (eg., belief, behavior and knowledge). I conducted CFA on this three measures using MPlus. My first objective is to check the validity using CFA (measurement models), and then the next stage is to run a SEM model (obj2) based on the result form the measurement models (obj1). In my obj 1, is it necessary to calculate the composite reliability and AVE (average variance extracted)? Fit indices from Mplus output for the measurement model indicate fit. However the AVE is quite low for some subscales/factors. I just wonder can I proceed to obj 2 (SEM model), even though my AVE is quite low for some factors (eg say 0.40 < recommended 0.50?). I have seen many articles using CFA analysis had only reported the fit indices and very rare author will report the composite reliability and AVE. My question is the fit indices provided in Mplus are sufficient enough to judges a measurement model is valid?
Cheng posted on Saturday, April 04, 2015 - 11:38 pm
Sorry, I have another 2 questions:
(1)Regarding the knowledge scale which is binary (yes/no). In Mplus, I run CFA using estimator WLSMV. The standardized correlation between two factors is 1.012 which is more than 1. What can I do, should I delete more items? Does it mean that these two factors are highly correlated? Can you use the word “correlated” for binary measure?
(2)If I want to calculate the composite reliability and AVE, what formula should I use for knowledge which is a binary measure? From your previous discussion, composite reliability should refer to Raykov et al 2010. How about AVE? Can we calculate AVE for binary measure?
Dear Muthen, I had read Raykov’s paper on “evaluation of scale reliability with binary measures using latent variable modeling”. The scale reliability coefficient formula stated in the paper (page 269, formula (14)), can it be applied if WLSMV estimator is used in CFA? Is there any way I can compute AVE (average variance extracted) for binary measure (yes and no response)?
I wanted to ask you 2 questions regarding reliability.
1) From reading earlier comments and your responses, I have noted an Mplus code for calculating Omega reliability for a latent factor with continuous indicators. I was wondering, if I have ordinal indicators measured on a 5-point Likert scale, would that be ok to use the same code or do I have to necessarily use the code you suggested for the categorical/binary indicators from: "Evaluation of Scale Reliability With Binary Measures Using Latent Variable Modeling"?
2) From your experience would the procedures for ordered categorical (e.g.5 point likert scale) and continuous indicators give very different or somewhat similar results?
3) Given the omega code for continuous indicators, how would one adopt this formulae for measuring a second-order latent factor omega that has 3 first order latent factors and 4 indicators per factor. I thought I could first calculate omegas for the 3 first order factors.But then I am not sure how to proceed next, i.e. how they should be combined and a second-order factor taken into account.