+
Skip to content

documentation: Clarify extraction of learner-specific outputs (e.g., coefficients, importance) after tuning with AutoTuner #1324

Open
@vstorme

Description

@vstorme

Hi,

As a user of mlr3 and mlr3learners, I’d like to suggest a documentation improvement for workflows that involve tuning models with AutoTuner.
While the tuning and resampling pipelines work very well, I found it difficult to access learner-specific outputs such as:

  • coefficients from regr.glmnet (Lasso/Ridge)
  • feature importance from regr.xgboost or regr.ranger

After tuning a learner using AutoTuner, I expected that I could extract model-specific details from at$model, but this either fails silently (e.g., coef() returns NULL) or throws errors (importance() unavailable). It took a lot of debugging to realize that the cleanest solution is to:

  • Extract the best hyperparameters from at$archive$best()
  • Reconstruct a new learner manually with fixed parameters
  • Train it directly on the task
  • Then access coef() or importance() safely from the learner-specific backend

I believe this pattern should be clearly documented in the mlr3book, perhaps under:

  • “Model Inspection” or
  • “Post-tuning workflow”

This would save users significant time, especially those expecting a caret- or tidymodels-like experience where coefficients or importances are easily accessible post-fit.

Thanks for all your great work — mlr3 is an incredibly powerful and well-designed framework, and I hope this suggestion helps smooth the learning curve a bit for others.
This is now my workflow for a lasso model:

`# 1/ Split into train/test (randomly, 80/20)
set.seed(1358)
splits = partition(tsk_mtcars, ratio = 0.8)

2/ define tuner

tnr_random_search = tnr("random_search")

3/ define terminator

trm_evals100 = trm("evals", n_evals=100)

4/ define learner

lrn_lasso = lrn("regr.glmnet",
alpha = 1, # Lasso
lambda = to_tune(p_dbl(lower = 1e-4, upper = 1, logscale = TRUE))
)

5/ Define inner and outer CV for nested resampling

outer_cv8 = rsmp("cv", folds = 8)
inner_cv5 = rsmp("cv", folds = 5)

6/ Define performance measure

msr_mse = msr("regr.mse")

7/ Wrap learner in AutoTuner

at = auto_tuner(
tuner = tnr_random_search,
terminator = trm_evals100,
learner = lrn_lasso,
resampling = inner_cv5,
measure = msr_mse
)

8/ Nested CV (on training set only):

first clone the task and then filter it (the resample function has no row_ids option)

tsk_mtcars_train <- tsk_mtcars$clone()$filter(splits$train)

set.seed(1358)
rr_lasso = resample(tsk_mtcars_train, at, outer_cv8, store_models = TRUE)

9/ Extract the generalization error and the standard error

rr_lasso$aggregate(msr_mse)
scores = rr_lasso$score(msr("regr.mse"))
mean_mse = mean(scores$regr.mse)
sd_mse = sd(scores$regr.mse)
se_mse = sd_mse / sqrt(dim(scores)[[1]])

10/ Inspect tuned hyperparameters for each fold (lambda is on the log scale)

extract_inner_tuning_results(rr_lasso)[, .(iteration, lambda, regr.mse)]

11/ Retrain best model on training data

set.seed(1358)
at$train(tsk_mtcars, row_ids = splits$train)
data.table(at$archive$data)[, .(lambda, regr.mse, runtime_learners)]

best hyperparam lambda:

lambda_log = at$archive$best()$lambda
as.data.table(at$archive)

12/ Predict on held-out test set

predict_test = at$predict(tsk_mtcars, row_ids = splits$test)
rmse_test = round(predict_test$score(msr_mse),2)

convert to data frame

tsk_mtcars$truth(splits$test)
pred.dt <- as.data.table(predict_test)
pred.df <- as.data.frame(pred.dt) #didn't work when used directly on R6 object

13/ plot truth vs predicted (response)

autoplot(predict_test)

14/ model coefficients of final model:

this requires to reconstruct the learner with the best parameters found in previous steps

the following does not return a glmnet object:

no_glmnet_model = at$model
class(no_glmnet_model)

"auto_tuner_model" "list"

14.1 extract best parameters after tuning:

best_params = at$archive$best()$x_domain[[1]]

14.2 Rebuild the final model with the best parameters

lrn_lasso_final = lrn("regr.glmnet",
alpha = 1,
lambda = best_params$lambda
)

14.3 train on the train data set

lrn_lasso_final$train(tsk_mtcars, row_ids = splits$train)
names(lrn_lasso_final)

14.4 extract coefficients

glmnet_model = lrn_lasso_final$model
class(glmnet_model)

"elnet" "glmnet"

coefs = coef(glmnet_model, s = best_params$lambda)
as.matrix(coefs)[as.matrix(coefs) != 0, , drop = FALSE]`

best regards,
Veronique

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      点击 这是indexloc提供的php浏览器服务,不要输入任何密码和下载