Jon Heron posted on Tuesday, September 04, 2012 - 2:08 am
thanks for a very inspiring few days in Utrecht.
There was a brief discussion last thursday on the topic of how to deal with missing data amongst covariates when you get to the third step of the 3-step approach. I meant to quiz you further on this but didn't get the chance.
I am about to submit a paper where I dump my data into a chained-equation imputation model prior to the "modal-ML" third step as a way of filling in gaps in the covariates. This seemed to work quite well.
I also have an applied paper in print where I tried imputation as the first step - this was a nightmare as I had to fit 100 mixture models and pool the results.
My sense is that this is far from an answered question and that there are still many interesting avenues to explore - would you agree? perhaps you already sort of know the answer. My reason for asking is that i'm toying with the idea of making a PhD project around this, combining simulations and real-life data.
Good to see you there, Jon; great meeting. Our current thinking is that you do step 1 and 2 as usual and only deal with missing on x in step 3, doing the missing on x by Mplus multiple imputation (perhaps using other variables to impute). So step 3 will be a Type=Imputation run over the different x-imputed data. It is a research topic as you say.
Jon Heron posted on Tuesday, September 04, 2012 - 7:02 am
Thanks Bengt, I'll have a play around with that.
On a semi-related note, I have just finished estimating a 3-step (C on X) model in Mplus v6 and can report that the model based on modal assignment agrees perfectly with Latent Gold - no surprise there!
I have also attempted to implement Vermunt's "Proportional ML" 3-step model which uses the probability weights and an alternative classification matrix. Here the estimates are great, but the SE's are attenuated as Mplus' "weight" command is rescaling the sample size so my assignment probabilities will no longer sum to 1 within respondent. I think you said there was a trick for this?
Looks good. You are certainly an "early adopter". Be sure to take the misclassification errors from a separate analysis for each latent class variable and not the joint one.
Jon Heron posted on Thursday, September 06, 2012 - 5:54 am
Thanks Bengt, good tip. These are both from separate already-published LCA models.
Something that comes out of Vermunt's paper that is a real surprise, though consistent with your own simulation study is that proportional assignment is actually *worse* than modal-class. We are having trouble convincing colleagues of this fact.
If you calculate Vermunt's D-matrix (the flipside of Mplus' classification table) for the two approaches then the one for modal-class is better in terms of the magnitude of the probabilities on the main diagonal compared with proportional assignment. I was initially shocked to discover this matrix was not actually the identity matrix when using proportional assignment, but you can't argue with the math.
Jon Heron posted on Friday, September 14, 2012 - 7:31 am
I've been having a play today - attempting to incorporate imputation prior to the final step of the 3-step approach.
Parameter estimates look believeable, and pleasingly similar to the complete case findings, however the p-values look too high
For instance, with a 4-cat exposure and a binary outcome, I obtain 3 odds ratios:-
but model test gives an omnibus p-value of 0.0513.
I have replicated the estimates in latent gold but obtained a lower p-value (0.0001). I have also removed the fancy 3-step bit from both programs and fitted simple logistic models - there is still a discrepancy in p-values.
Incidentally, without the imputation, both packages agree totally, so it's not a mistake in my model-test statement.
I know we need to be wary of significance testing when working with imutation - you can't pool LR-tests - but it was my understanding that the Wald was OK.
Jon Heron posted on Friday, September 14, 2012 - 11:40 am
OK, I'll try and replicate with a small example so I can send.
Of course that usually leads to a solution in itself
Sar posted on Wednesday, September 26, 2012 - 4:21 am
Hi, I am trying to manually compute the error rate for N using the output "Average Latent Class Probabilities for Most Likely Latent Class Membership by Latent Class" in Step 2 of the 3-Step Mixutre Modeling method. On slide 53 of the Mplus 7 version training handouts (New Developments in Mplus Version 7: Part 2) there is an example for a 2-Class model and I was just wondering how to do this for a 4-Class model.
Also I just had a question about using AUXILIARY=x(R3STEP) with multiple predictors- when you look at main effects for each predictors does this covary for other predictors? Thank you
Going from 2 to more classes simply means that you have to compute more log odds of the kind shown on the next slide, using the values of the classification table ("Average Latent Class Probabilities...}. So compute log of the value for a certain class divided by the last class - with 4 classes this gives 3 such log odds. You do this for every row (every class). And your 3rd step will then have 4 classes instead of 2.
The R3STEP procedure gives you a multinomial logistic regression for the covariates included in the auxiliary list. So in line with regular regression, each coefficient is a partial regression coefficient showing the effect of one covariate holding the other covariates constant.
A revised version of the Mplus Web Note 15 with more extensive instructions will be posted shortly.
Jon Heron posted on Thursday, September 27, 2012 - 11:58 pm
This 3-step approach where we correct for the classification error in a categorical manifest variable has got me thinking about another potentially interesting application.
I've recently published an LCA model of repeated measures of self-reported smoking frequency. (there are now a fair few of those!!).
We've since acquired some measures of cotinine taken from blood samples. I'm wondering about the possibility of using this additional information to get an idea of the accuracy of the self-report and then using this to adjust the manifest measure.
Unlike the 3-step which is based on nominal measures, this would be ordinal, but it feels to me like it might be a go-er.
Not really a question that, just an early-morning ponderance.
I like the idea of trying to get a better understanding of the measurement error. What would you say to a Devil's advocate who would want include the cotinine measures in the first step?
Jon Heron posted on Friday, September 28, 2012 - 9:09 am
We don't have cotinine assessed at each wave and it may be a year or so before we have it for more than one wave. Hence I envisaged a one-time point comparison that would then inform the next stage of modelling, based on the (dubious?) assumption that self-report accuracy is constant.
My initial hope was we would have repeated pairs of self-report and biological samples and to do a one-step. Is that enough our a reason not to?
Sar posted on Wednesday, October 24, 2012 - 2:10 pm
Hi, I just wanted to confirm that I am correctly computing the error rate for N using the output "Average Latent Class Probabilities for Most Likely Latent Class Membership by Latent Class" in Step 2 of the 3-Step Mixutre Modeling method. My main concern is what do I do when I am taking the log of 0, or in the special case where the last class has 0. Otherwise it makes sense, it is just in these cases I am unsure what to do. I have copied a section of my output and the calculations for the first 2 classes below. Thank you so much.
Average Latent Class Probabilities for Most Likely Latent Class Membership (Row) by Latent Class (Column)
Alternatively in your computations above replace the 0 probability with 0.0001.
Lewina Lee posted on Sunday, April 07, 2013 - 8:25 pm
I am trying to compare classes on multiple distal outcomes, which have significant missingness. I attempted multiple imputation and realized that MI only works with R3STEP but not DU3STEP. Is there a workaround to this?
I noticed the earlier posts by Jon Heron addressing perhaps a similar issue. If the manual 3-step can be implemented along with MI, at which step do I perform MI?
If you want to do this using multiple imputation, you must do it for each data set and then correctly combine the results. If you can use FIML, it would be a lot easier for you.
Lewina Lee posted on Tuesday, April 09, 2013 - 7:31 am
When you suggested using FIML to handle missingness in class predictors and distal outcomes, did you mean outputting the most likely class assignment for each person, and doing the analysis (e.g., C on X) with a regression approach? I have an entropy of .80.
No, I meant instead of using multiple imputation to impute missing values to use TYPE=MISSING which is the default in Mplus.
Lewina Lee posted on Tuesday, April 09, 2013 - 2:52 pm
Thank you, Linda. I can see that FIML can handle missingness on distal outcomes.
When auxiliary variables used in R3STEP are missing, they are treated as exogenous variables and so FIML doesn't apply (i.e., listwise deletion on cases missing Xs). Is there a way to "trick" M+ to treat them as endogenous in the mixture model?
I have a similar issue as above. I have a few hundred missing cases on distal outcomes and 1 covariate I want to use in a latent profile analysis using the BCH method. If I use either the outcome or covariate in the analysis, there is listwise deletion of all cases missing on x or the distal outcome before fitting the model right? Or would FIML apply? If the only approach to retaining the hundreds of observations is either MI or lw-deletion, how would I go about imputing? You mentioned above that it would be a daunting task. Thanks!
Jon Heron posted on Thursday, January 28, 2016 - 11:39 pm
In principle FIML would apply for your distal outcome, however, while I think FIML can be justifiable when considering missingness in one of many class indicators, I think the MAR assumption can be more questionable for a distal outcome subsequent to you latent class model. (Class indicators are excellent predictors of other class indicators).
I've had a recent unsuccessful grant which had the aim of addressing the missing covariate problem and I'm currently re-working this for resubmission. There have been a number of pragmatic solutions adopted in the applied literature. These may satisfy reviewers but perhaps not the bias-police. Ultimately the solution is probably Bayesian but this is likely to rule out the more complex problem, and the less-hardened analyst.
Of course your adopted method should depend on your missing data assumptions. Have you considered sticking with the listwise deletion of covariates and employing IPW (probability weighting) to re-adjust your sample?