I have data in which classes are assigned from within schools to two treatments. Scores are child-level. I fit a latent means ANCOVA model. BIC and adjusted BIC are smallest for a model with no level-3 random effects: MODEL: %within% ep_pst on ep_pre; %between tchid% TX_ep2|ep_pst on trt; TX_ep1|ep_pre on trt; ep_pst on ep_pre; %between schid% ep_pst on site; ep_pre on site; TX_ep1 on site; TX_ep2 on site; TX_ep1@0; TX_ep2@0; ep_pre@0; ep_pst@0; ep_pst with ep_pre@0;
I also fit a two-level model
MODEL: %within% ep_pst on ep_pre; %between% ep_pst on trt site TxS; ep_pre on trt site TxS; ep_pst on ep_pre;
-2LL is identical for the two models. There are two notable differences between results for the two models: the intercept and the site effect for ep_pst. The intercept difference is due to centering differences. But I do not understand the source of the site effect difference. Can you explain why the difference occurs?
The following may be of interest: I have tried the two models with other variables from the same design and have three examples in which the fit statistics are different for the constrained 3-level model and the 2-level model. In each case it was necessary to use more than 500 miterations in the 3-level model to avoid this warning "AN INSUFFICENT NUMBER OF E STEP ITERATIONS MAY HAVE BEEN USED." In each example fit statistics are better for the 2-level model.
To review since I was not able to reply earlier this week: the constrained 3-level model does not have any level-3 variance or covariance parameters.
By "that's an important finding", do you mean the finding that fit statistics (AIC, BIC, and sample size adjusted BIC) are better for the 2-level model or do you mean the finding of significant level 3 parameter estimates?
There are parameters (a site effect and an intercept) with significant estimates at level 3, but these parameters also have significant estimates at level 2. And some parameters that have estimates that are non-significant at level 3 have estimates that are significant at level 2. Overall I have interpreted the better fit statistics for the 2-level model to mean I can interpret its results. Is this reasonable?