AIC/BIC with different estimated models
Message/Author
 Dorothea Müller posted on Friday, October 18, 2013 - 8:27 am
I’m comparing a mediator model (type= general) with a moderator model, which requires numerical integration (type=random; algorithm= integration). Am I right in assuming that the AIC/BIC of the two models are not comparable? And if yes, why and is there any literature about this topic?
 Bengt O. Muthen posted on Friday, October 18, 2013 - 8:43 pm
If you have the same DVs in the two runs the AIC/BIC values are comparable.
 Dorothea Müller posted on Monday, October 21, 2013 - 8:01 am
Thanks for your quick response. My dependent variables are the same:
1. Mediator-Model:
Analysis: type=general
Model: X by X1.1. X1.2. X1.3. X1.4. X1.5
Y by Y1.1. Y1.2. Y1.3. Y1.4.
Y on Z C X
X on Z
X on C
Model indirect: Y ind Z; Y ind C;
2. Moderator-Model:
Analysis: type: random; algorithm= integration
X by X1.1. X1.2. X1.3. X1.4. X1.5
Y by Y1.1. Y1.2. Y1.3. Y1.4.

ZxX | Z xwith X;
CxX | C xwith X;
Y on X Z C ZxX CxX;

The mediator model has significant effects whereas the moderator model has no significant interaction-effects – nevertheless BIC and AIC suggest the first model. I’m aware that the information criterions have nothing to do with significance, even so I’ m still afraid that I’m missing something regarding AIC/BIC.
 Bengt O. Muthen posted on Monday, October 21, 2013 - 5:01 pm
The model with interactions (moderation) has extra parameters and those are not significant. That means that BIC is worse for this model because the likelihood is not improved enough to compensate for the extra parameters. So the results make sense to me.
 Margarita  posted on Friday, March 27, 2015 - 7:52 am
Dear Dr. Muthén,

I wanted to clarify something about the AIC. Can it be used to compare models with different parameters? Or do they need to have at least the same number of parameters to be comparable?

Thank you!
 Bengt O. Muthen posted on Friday, March 27, 2015 - 9:03 am
Q1 Yes.

Q2. No. They need to have the same DVs.
 Margarita  posted on Friday, March 27, 2015 - 9:07 am
Thank you very much!
 Margarita  posted on Tuesday, September 15, 2015 - 10:22 am
Dear Dr. Muthen,

I have a followup question regarding AIC.

You previously said that AIC can be used to compare models as long as they have the same DVS. In mediation models, given that the mediators are considered to be DVs, I would not be able to compare models with different mediators correct?

For example comparing a simple mediation model :
X -> M -> Y

to a parallel model: X->M1, M2-> Y

If I understand correctly, the competing models need to contain the exact same number and type of DVs?

Thank you!
 Bengt O. Muthen posted on Tuesday, September 15, 2015 - 1:19 pm
Right.
 JHS posted on Wednesday, February 24, 2016 - 5:56 pm
Can AIC/BIC be compared when the IVs and DVs are reversed in two models?

Ex.

Model 1: X-> M -> Y
Model 2: Y-> M -> X

Thank you!
 Bengt O. Muthen posted on Wednesday, February 24, 2016 - 6:19 pm
No, you have to have the same DV for the metric to be comparable.
 JHS posted on Thursday, February 25, 2016 - 12:42 pm
Thank you. Is there a way to compare model fit in the above scenario then? (using all the same variables in each model)
 Bengt O. Muthen posted on Thursday, February 25, 2016 - 6:23 pm
I am not aware of it.
 Katrin Lintorf posted on Friday, November 11, 2016 - 3:13 pm
This is a follow-up-question to Margaritas post on September 15th, 2015: I understand that I can't compare models with different DVs, e.g. in case of a replaced DV. Do I also have models with different DVs in case of an added DV? - I was asked to compare two models: a model without mediator (Y on X) and a model with the added mediator (Y on X M; M on X). In case that this is relevant: I have a two-level model with type=imputation and a dichotomous Y.

Kind regards
Katrin
 Bengt O. Muthen posted on Friday, November 11, 2016 - 4:45 pm
Yes, these types of analyses also cannot be compared via AIC/BIC.
 Katrin Lintorf posted on Sunday, November 13, 2016 - 1:53 pm
Thank you. Could you please give me the reason, why the analyses are not comparable via AIC/BIC - and do you have a reference for the explanation? Unfortunately, I was not successful in finding a corresponding reference.
Kind regards
Katrin
 Bengt O. Muthen posted on Sunday, November 13, 2016 - 6:27 pm
The loglikelihood is on different scales unless the two models have the same DVs. I don't know of references - it is a basic fact.
 Hannah Helfrich posted on Tuesday, December 13, 2016 - 5:37 am
Dear Dr. Muthen,

I am comparing a (latent) mediation model and a moderated mediation model. Two mediators (LEH, AS) operate in parallel. The interaction in the moderated mediation model only affects the path from the predictor to one mediator (LEH).
To give you more information, see my code and the fit indices below:

Model WITHOUT interaction:
ANALYSIS:
Type = general;
estimator = ML;
MODEL:
Trust by m2_tru01 m2_tru02 m2_tru03 m2_tru04;
IAT by iat_h1_p iat_h2_p iat_h3_p;
LEH by lehpar01 lehpar02 lehpar03;
AS by aspar01 aspar02 aspar03 aspar04;

Trust on LEH AS;
LEH on IAT;
AS on IAT;
LEH with AS;

Free Parameters: 92
Loglikelihood H0 Value: -4670.967
AIC: 9525.933
BIC: 9849.908

Model WITH interaction:
ANALYSIS:
Type = random;
estimator = ML;
Algorithm = integration;
MODEL:
Trust by ...;
...

IATSex | IAT XWITH f_sex;
Trust on LEH AS;
LEH on IATSex;
AS on IAT;
LEH with AS;

Free Parameters: 94
Loglikelihood H0 Value: -4823.946
AIC: 9835.893
BIC: 10166.910

Do you have an idea why the model fit indices in the model with interaction are clearly worse than those in the model without interaction although the interaction is significant?
And how can you interpret this fact when comparing the two models?
 Bengt O. Muthen posted on Tuesday, December 13, 2016 - 6:19 pm
In the interaction model you forgot

LEH on IAT;

There should be only a difference of 1 in the number of parameters.
 Hannah Helfrich posted on Wednesday, December 14, 2016 - 1:21 am