jeon small posted on Friday, March 01, 2013 - 8:05 pm
How do you perform a posthoc power analysis. I have a sample of 438 girls and boys. We used 11 indicator variable to form four variables. The fit of the structural model was satisfactory with CFI=.954, RMSEA=.05, DF=40 CHI SQUARED=95.85 PROBABILITY=.0000 The standardized regression weights were significant and ranged in size form .65-.03.
See Example 12.7 where the values from a real data analysis are saved and used as population parameter values for a Monte Carlo simulation study that can examine power. This method has its critics. You may want to explore this issue on SEMNET.
jeon small posted on Monday, March 04, 2013 - 1:38 pm
I ran this model in Amos using maximum-likelihood estimation (for the missing data)--however, I can not get the model to converge in Mplus. What am I doing wrong? My output is below.
STRUCTURAL MODEL OF ADOLESCENT ALCOHOL USE AND PERCEPTION OF IPV
SUMMARY OF ANALYSIS
Number of groups 1 Number of observations 553
Number of dependent variables 10 Number of independent variables 0 Number of continuous latent variables 4
Estimator ML Information matrix OBSERVED Maximum number of iterations 1000 Convergence criterion 0.500D-04 Maximum number of steepest descent iterations 20 Maximum number of iterations for H1 2000 Convergence criterion for H1 0.100D-03
Input data file(s) h:/M_russell/mplus10_08_12v1.csv
I would like to estimate the power to detect that a path coefficient is different from zero and am trying to use https://www.statmodel.com/power.shtml as a procedure. Alas it hasn't worked so far. A simplification of my model looks like the following, with three IVs and three DVs. Two of those are latent, four manifest:
IV1 by Indicator1 Indicator2 Indicator3;
DV1 by Indicator4 Indicator5 Indicator6;
DV1 on IV1 IV2 IV3;
DV2 on IV1 IV2 IV3;
When I follow step 1 of the procedure on https://www.statmodel.com/power.shtml, fixing values for the factor loadings and the path coefficients, the resulting covariance matrix of the residuals contains zeros, which is not accepted by Mplus in step 2. Do I have to fix more in step 1 than the factor loadings and the path coefficients?
(In step 1, I have used the most simple matrix possible, in the form of: 0 0 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 )
I have a question concerning the interpretation of a post-hoc power analysis following the procedure described in your 2002 paper. Suppose I have a population estimate for a specific path of 0.345 and the Monte Carlo Simulation gives a standard error average of 0.10, and a "%Sig Coeff" of 0.950. Can I divide the estimate by the standard error average to obtain a z-value, transform this into d based on my sample size (say, N=200), and finally conclude that my study had 95% power to detect a medium sized path? Or how would the results of this simulation be translated into "conventional post-hoc power analysis language"?
I don't know what "d" is but in general, it is better to express in Model Constrain the quantity you are interested in and get the power for it the regular way under New parameters.
Lara Ditrich posted on Tuesday, February 11, 2020 - 12:11 am
Thank you for your quick reply and sorry for not being clear enough in my previous post. With "d" I meant Cohen's d as an effect size estimate.
Just to make sure I understand you correctly: I am using the parameters from my model as population estimates for the Monte Carlo simulation. Your suggestion is to define a new parameter of a certain size under "model constraint" (e.g., y on x@some value) that is not part of the original model and look at the "%Sig Coeff" to determine how large the power was to detect this effect, right?
If this is right, I have two follow-up questions: Should I define this parameter only in the "Model"-part of the MC simulation or also in the "Model Population" part? And how do I know whether this parameter is "small", "medium", or "large" in size in terms of conventional rules of thumb?
Cohen's d is (m1-m2)/sd, where the m's are means for two different conditions and sd is the relevant standard deviation. You can express d that way in Model Constraint and thereby get the power for the d. I don't know, however, what your two conditions are.
Lara Ditrich posted on Wednesday, February 12, 2020 - 12:26 am
Thank you again for your helpful reply. It made me realize that expressing the effects as ds does not make much sense in my case, as I don't have a condition variable in my path model.
I assume it would be more appropriate to use r as an effect size in my case and then to include for example
I would simply choose parameter values that make the variance of Y and X one. Then Y ON X will be in a standardized metric where I think the Cohen standard for small, medium, large effects would be relevant.
If I understand your reply correctly, this means that I should not use the parameter estimates from my model as input for the MC simulation, but should generate a mean vector and covariance matrix that ensures the variances of my variables are 1 as the input for Step 2. Is that correct?
P.S.: Just to be sure - does your reply mean that I can interpret the STDYX estimates in terms of Cohen's standards for d?
Sorry for posting again. I just wanted to check back with you whether a solution I came up with is in line with what you suggest: I run Step 1 with z-standardized variables (which makes all their variances 1) and then feed the estimates into Step 2. I then interpret the column "%Sig Coeff" as my study's power to detect an effect of the estimate's size (which represents a beta-weight).