From 849e21925ea1c25bcab059ab11055ce21654e7a5 Mon Sep 17 00:00:00 2001 From: Sebastian Fischer Date: Mon, 16 Jun 2025 10:01:33 +0200 Subject: [PATCH 1/3] add warning that validation scores of graphlearner are prefixed by pipeop id --- book/chapters/chapter15/predsets_valid_inttune.qmd | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/book/chapters/chapter15/predsets_valid_inttune.qmd b/book/chapters/chapter15/predsets_valid_inttune.qmd index 0e3f7a386..2372dd532 100644 --- a/book/chapters/chapter15/predsets_valid_inttune.qmd +++ b/book/chapters/chapter15/predsets_valid_inttune.qmd @@ -281,7 +281,7 @@ Internally, this will train XGBoost with 10 different values of `eta` and the `n When combining internal tuning with hyperparameter optimization via `r ref_pkg("mlr3tuning")` we need to specify two performance metrics: one for the internal tuning and one for the `Tuner`. For this reason, `mlr3` requires the internal tuning metric to be set explicitly, even if a default value exists. There are two ways to use the same metric for both types of hyperparameter optimization: -1. Use `msr("internal_valid_scores", select = )`, i.e. the final validation score, as the tuning measure. As a learner can have multiple internal valid scores, the measure allows us to select one by specifying the `select` argument. We also need to specify whether the measure should be minimized. +1. Use `msr("internal_valid_scores", select = )`, i.e. the final validation score, as the tuning measure. As a learner can have multiple internal valid scores, the measure allows us to select one by specifying the `select` argument. If this is not specified, the first validation measure will be used. We also need to specify whether the measure should be minimized. 2. Set both, the `eval_metric` and the tuning measure to the same metric, e.g. `eval_metric = "error"` and `measure = msr("classif.ce")`. Some learners even allow to set the validation metric to an `mlr3::Measure`. You can find out which ones support this feature by checking their corresponding documentation. One example for this is XGBoost. The advantage of using the first option is that the predict step can be skipped because the internal validation scores are already computed during training. @@ -302,6 +302,10 @@ ti = tune( ) ``` +:::{.callout-warning} +When working with a `r GraphLearner`, the names of the internal validation scores are prefixed by the ID of the corresponding `r PipeOp`, so the `select` parameter needs to be set to `"."`. +::: + The tuning result contains the best found configuration for both `eta` and `nrounds`. ```{r predsets_valid_inttune-029} From 2573a124fa7d0fb91da1531508ab5d0255fee766 Mon Sep 17 00:00:00 2001 From: Sebastian Fischer Date: Mon, 16 Jun 2025 11:44:51 +0200 Subject: [PATCH 2/3] Update book/chapters/chapter15/predsets_valid_inttune.qmd --- book/chapters/chapter15/predsets_valid_inttune.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/chapters/chapter15/predsets_valid_inttune.qmd b/book/chapters/chapter15/predsets_valid_inttune.qmd index 2372dd532..495d8dd1d 100644 --- a/book/chapters/chapter15/predsets_valid_inttune.qmd +++ b/book/chapters/chapter15/predsets_valid_inttune.qmd @@ -302,7 +302,7 @@ ti = tune( ) ``` -:::{.callout-warning} +::: {.callout-warning} When working with a `r GraphLearner`, the names of the internal validation scores are prefixed by the ID of the corresponding `r PipeOp`, so the `select` parameter needs to be set to `"."`. ::: From 61df0c8570656fcf879531d1483db9576961bd2b Mon Sep 17 00:00:00 2001 From: Sebastian Fischer Date: Mon, 16 Jun 2025 11:45:59 +0200 Subject: [PATCH 3/3] Update book/chapters/chapter15/predsets_valid_inttune.qmd --- book/chapters/chapter15/predsets_valid_inttune.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/chapters/chapter15/predsets_valid_inttune.qmd b/book/chapters/chapter15/predsets_valid_inttune.qmd index 495d8dd1d..5a1721a94 100644 --- a/book/chapters/chapter15/predsets_valid_inttune.qmd +++ b/book/chapters/chapter15/predsets_valid_inttune.qmd @@ -303,7 +303,7 @@ ti = tune( ``` ::: {.callout-warning} -When working with a `r GraphLearner`, the names of the internal validation scores are prefixed by the ID of the corresponding `r PipeOp`, so the `select` parameter needs to be set to `"."`. +When working with a `GraphLearner`, the names of the internal validation scores are prefixed by the ID of the corresponding `PipeOp`, so the `select` parameter needs to be set to `"."`. ::: The tuning result contains the best found configuration for both `eta` and `nrounds`.