Regarding differences in raw AICc values generated by MuMIN and model_performance function from performance package

[nikhilm2510]

Dear authors, I observed several important differences when calculating raw AICc values using the MuMIn and performance packages. First, I would like to set the context. While comparing the performance of my models, I received an output with absolute confidence in one model (AICc weight = 1.0), and others had a value of 0.0. I was therefore interested to know the raw AICc values, which I tried to derive from the MuMIN package, and to my surprise, the same model with an AICc weight of 1.0 did not have the lowest AICc value with MuMIN; in fact, it was one of the mid-range values (raw AICc = 2312.7498). I was then motivated to check the same with the model_performance function from the performance package, and it came out to be -3814.8, which is way too different from the earlier value. Is it because the model_performance function is using a different formula for calculating AICc values than MuMIN? Interestingly, the models with the lowest AICc values in MuMIN had identical values to model_performance, which is quite striking.

models that I compared

[mattansb]

Can you provide a reproducible example?

[nikhilm2510]

If I am not wrong, this could also be because of log transformed response variable. In fact, model_performance warns about the inaccuracy in handling log-transformed response (even after REML = FALSE). In such a case, wouldn't it be risky to use compare_performance for model comparison?.

[bwiernik]

To add some more detail for clarity:

1. AIC functions in base R and most packages take the response variable in a model as given and do not consider any transformations that are applied. This means that models with different transformations of the response variable cannot be compared (eg, no transformation vs log transformation). This can be a source of confusion for many R users who inadvertently draw erroneous conclusions by comparing models with different transformations. It is also limiting for users who wish to accurately compare performance of models with different transformations.

2. The performance package avoids this problem by adjusting the log likelihood when possible to return the values to the scale of the original variable. This works well for transformations such as log() or sqrt(). For a small number of transformations, namely inverse (1/y), the jacobian adjustment doesn't work but this is a niche case.

3. The adjustment for log() is an additive constant, so the rank order (and aic differences) of models using the same transformation and the same data will be unchanged. If this isn't case, the most likely explanation is that the response data are different for the models, such as due to different missing data for predictors.

If that isn't the behavior you are seeing, can you please provide a reproducible example so that we can investigate?

[bwiernik]

@strengejacke I notice that performance always defaults to REML = TRUE for lme4 models. It should probably default to what is used for the estimation?

[strengejacke]

@strengejacke I notice that performance always defaults to REML = TRUE for lme4 models. It should probably default to what is used for the estimation?

good point! I'll check.

[strengejacke]

@bwiernik performance_aic() is alerting the user (clever package 😎 ):

m1 <- lme4::lmer(Sepal.Length ~ Petal.Length + (1 | Species), data = iris)
performance::performance_aic(m1)
#> [1] 127.793

m2 <- lme4::lmer(Sepal.Length ~ Petal.Length + (1 | Species), data = iris, REML = FALSE)
performance::performance_aic(m2)
#> Model was not fitted with REML, however, `estimator = "REML"`. Set
#>   `estimator = "ML"` to obtain identical results as from `AIC()`.
#> [1] 128.0054

^{Created on 2026-03-10 with reprex v2.1.1}

[bwiernik]

Any reason to warn vs doing what was done in estimation as default?

[nikhilm2510]

Here is a snippet of a working example. Firstly, I kept all REML = FALSE before model comparison. Although model_performance asks to set estimator = ML, in such a case, it might not be required, as REML = FALSE alternatively means the estimator = ML. Further, the warning "....this ignores REML = TRUE looks unnecessary when REML = FALSE. Then, in such a case (when REML = FALSE), should I consider that the log-likelihood attempted by model_permornance is accurate and reliable? If that is so, would it be fair to say that the results of the
performance package are better than MuMIn?

growth_rate_for_github.docx