Description
Hi,
As a user of mlr3 and mlr3learners, I’d like to suggest a documentation improvement for workflows that involve tuning models with AutoTuner.
While the tuning and resampling pipelines work very well, I found it difficult to access learner-specific outputs such as:
- coefficients from regr.glmnet (Lasso/Ridge)
- feature importance from regr.xgboost or regr.ranger
After tuning a learner using AutoTuner, I expected that I could extract model-specific details from at$model, but this either fails silently (e.g., coef() returns NULL) or throws errors (importance() unavailable). It took a lot of debugging to realize that the cleanest solution is to:
- Extract the best hyperparameters from at$archive$best()
- Reconstruct a new learner manually with fixed parameters
- Train it directly on the task
- Then access coef() or importance() safely from the learner-specific backend
I believe this pattern should be clearly documented in the mlr3book, perhaps under:
- “Model Inspection” or
- “Post-tuning workflow”
This would save users significant time, especially those expecting a caret- or tidymodels-like experience where coefficients or importances are easily accessible post-fit.
Thanks for all your great work — mlr3 is an incredibly powerful and well-designed framework, and I hope this suggestion helps smooth the learning curve a bit for others.
This is now my workflow for a lasso model:
`# 1/ Split into train/test (randomly, 80/20)
set.seed(1358)
splits = partition(tsk_mtcars, ratio = 0.8)
2/ define tuner
tnr_random_search = tnr("random_search")
3/ define terminator
trm_evals100 = trm("evals", n_evals=100)
4/ define learner
lrn_lasso = lrn("regr.glmnet",
alpha = 1, # Lasso
lambda = to_tune(p_dbl(lower = 1e-4, upper = 1, logscale = TRUE))
)
5/ Define inner and outer CV for nested resampling
outer_cv8 = rsmp("cv", folds = 8)
inner_cv5 = rsmp("cv", folds = 5)
6/ Define performance measure
msr_mse = msr("regr.mse")
7/ Wrap learner in AutoTuner
at = auto_tuner(
tuner = tnr_random_search,
terminator = trm_evals100,
learner = lrn_lasso,
resampling = inner_cv5,
measure = msr_mse
)
8/ Nested CV (on training set only):
first clone the task and then filter it (the resample function has no row_ids option)
tsk_mtcars_train <- tsk_mtcars$clone()$filter(splits$train)
set.seed(1358)
rr_lasso = resample(tsk_mtcars_train, at, outer_cv8, store_models = TRUE)
9/ Extract the generalization error and the standard error
rr_lasso$aggregate(msr_mse)
scores = rr_lasso$score(msr("regr.mse"))
mean_mse = mean(scores$regr.mse)
sd_mse = sd(scores$regr.mse)
se_mse = sd_mse / sqrt(dim(scores)[[1]])
10/ Inspect tuned hyperparameters for each fold (lambda is on the log scale)
extract_inner_tuning_results(rr_lasso)[, .(iteration, lambda, regr.mse)]
11/ Retrain best model on training data
set.seed(1358)
at$train(tsk_mtcars, row_ids = splits$train)
data.table(at$archive$data)[, .(lambda, regr.mse, runtime_learners)]
best hyperparam lambda:
lambda_log = at$archive$best()$lambda
as.data.table(at$archive)
12/ Predict on held-out test set
predict_test = at$predict(tsk_mtcars, row_ids = splits$test)
rmse_test = round(predict_test$score(msr_mse),2)
convert to data frame
tsk_mtcars$truth(splits$test)
pred.dt <- as.data.table(predict_test)
pred.df <- as.data.frame(pred.dt) #didn't work when used directly on R6 object
13/ plot truth vs predicted (response)
autoplot(predict_test)
14/ model coefficients of final model:
this requires to reconstruct the learner with the best parameters found in previous steps
the following does not return a glmnet object:
no_glmnet_model = at$model
class(no_glmnet_model)
"auto_tuner_model" "list"
14.1 extract best parameters after tuning:
best_params = at$archive$best()$x_domain[[1]]
14.2 Rebuild the final model with the best parameters
lrn_lasso_final = lrn("regr.glmnet",
alpha = 1,
lambda = best_params$lambda
)
14.3 train on the train data set
lrn_lasso_final$train(tsk_mtcars, row_ids = splits$train)
names(lrn_lasso_final)
14.4 extract coefficients
glmnet_model = lrn_lasso_final$model
class(glmnet_model)
"elnet" "glmnet"
coefs = coef(glmnet_model, s = best_params$lambda)
as.matrix(coefs)[as.matrix(coefs) != 0, , drop = FALSE]`
best regards,
Veronique