Peter Croy posted on Friday, August 04, 2006 - 1:31 am
Dear Mplus/SEM users, I have applied a benchmark model to my data. This model is based on theory. The model has 3 latent predictors with their respective indicators and a further latent (criterion) variable with indicators. I wish to add predictor variables to determine whether an expended model does a better job in accounting for variance in the criterion variable. 1. Am I correct in assuming that if I use modindices to achieve acceptable fit in the benchmark model, then the additive effects of the new predictor variable(s) may be masked by any respecifications (the WITH associations in particular) applied to the benchmark model? If so, should I leave the benchmark model as is (i.e., not respecified for possible systematic error) so that I can clearly discern effects of additional predictor variables? 2. What is the process for comparing one model to another? Do I simply compare Chi Sq values or R Sq ... or what? Many thanks
Adding predictors should in principle not affect the relations among the other variables in the model, except by having more power. But adding predictors can affect the fit of the model because there may be need for direct effects of predictors on indicators, not only on the factors. I would use chi-square to compare models.
Peter Croy posted on Friday, August 04, 2006 - 10:46 pm
Thanks Bengt ... begging your further indulgence, 1.I'm just unsure of whether the uniqueness (residuals) in the present model may represent systematic bias that could be accounted for when I add further latent predictors to the model. Can you just elaborate a little on why this gut-feel is incorrect? 2. Can you elaborate a little on what you mean by "direct effects of predictors on indicators"? 3. How would I test for significance of the difference in chi square when comparing one model to another? Further, how can I quantify chi square difference in a similar manner to which R sq. difference quantifies additive explanatory effects on the criterion variable in regression analysis? Thanks again.
1. Direct effects from the same predictor (covariate) to say 2 indicators could account for components of their residuals that are correlated across thoses 2 indicators. So that relates to your bias question.
2-3. Big topic so I will just refer to the treatment of direct effects in our Mimic-oriented papers and teachings. See Mimic papers on the web site, or Day 1 and Day 3 of our 5-day course. That also includes chi-square difference testing matters.
Peter Croy posted on Wednesday, August 23, 2006 - 4:31 am
Hi Bengt, I'm still struggling a bit - maybe your course notes would help. Can the notes be purchased and downloaded rather than airmailed?
Can I ask a simple question or 2 (or 3) about model fit ... I know you have plenty of other things to do but I'm sure your answers will be of great help to me ahead of any further reading on my part.
In my theory-based model I get significant chi-square results (p<.0001), but other fit indices are okay: CFI at .96 or thereabouts; RMSEA at .06 approx. My sample size is 2300. I had to correlate some residuals of indicators of a couple of latent variables to achieve these fit stats. and I have some logic for doing so. I interpret the significant chi-square result as being associated with sample size--i.e., I cannot expect to achieve non-significance with this sample size ... BUT, I have doubts about this because I obtained a good number of cross loadings when I ran an EFA at the outset. I would be interested to hear how you would interpret my model fit resuts? By the way, I also ran the model using MLM and achieved better fit indices but still a significant chi-square (Bar charts show as certain amount of polarisiation of responses on the likert scales).
I include two other questions under separate cover as I have exceeded allowed size.
Peter Croy posted on Wednesday, August 23, 2006 - 4:34 am
Bengt, Here's the second question, third to follow ... many thanks
Second, my project requires that I test the influence of variables (some latent with indicators and some dichotomous), which I introduce additional to the theory-based-model variables. Much past research in my field has used regression to test for significant incremental R-square effects when introducing independent (predictor) variables additional to the (parsimoneous) set of theory based variables. Assuming that I am not yet concerned with moderation, mediation or interaction, and I simply want to run the model with a single additional latent predictor, how do I asses the fit of the new model with the old theory based model in a single group context? If I can't rely on chi-square, what do I compare? Is AIC difference the way to go?
Peter Croy posted on Wednesday, August 23, 2006 - 4:36 am
Third, when I run multi-group (grouping is ...)models for differences between demographic groupings using Mplus default constraints I again obtain significant chi-square but other stats are more indicative of poor fit, CFI .92 approx and RMSEA .75 approx. In the absence of the issues I have with the underlying model (i.e., my issues as above) I would say that there is group non invariance. Can you tell me whether I should disregard the multi-group significant chi square result according to the same logic that I used in the case of assessing fit of the single group (underlying) model? OR, is this chi square (difference) statistic independent of the sample size issues inherent in my underlying model? If chi square (difference) still has sample size issues, how confident, can I be about group non invariance based on the values of the other fit statistics-i.e., .92 and .75? If I can conclude group non invariance, can you give me an idea of where to look for the source of the non invariance—i.e., is it standard practice to then go ahead and test specific parameter non invariance by adding/lifting model constraints?
Just a note in the positive spirit of keeping Mplus Discussion functioning as well possible - the size limitation for Mplus Discussion posts is there for a good reason, namely to promote specific, to-the-point questions related to Mplus modeling. So splitting questions up into parts is not in that spirit. These 3 posts raise analysis strategy questions which are too general to be practical for Mplus discussion - we take many hours to discuss these issues in our teachings. Although we wish we could do more to help researchers, we don't have the possibility to get into individual tutoring or consulting in this forum. So I would recommend two things (1) read the introductory SEM literature - we suggest many good such books in our teachings (handouts can be purchase off the web site), and (2) come to Day 1 in the upcoming Baltimore training session in October described on our web site.
Peter Croy posted on Thursday, August 24, 2006 - 3:23 am
Hi Bengt, I'll keep it short! I'd love to come to Baltimore ... but do you run Mplus training in Australia? One of my issues is that Mplus is not on the map down here--as far as I'm aware. Can the course notes be downloaded?
We have not yet taught in Australia although we have many users there. The questions you ask are not however specific to Mplus. You should be able to get the training to answer them in Australia. Also, we have free online videos.