confusing error messages with auto tuner #416
Description
hi @sebffischer
I ran this code:
mtask <- mlr3::tsk("mtcars")
Tlrn <- mlr3torch::LearnerTorchMLP$new(task_type="regr")
mlr3::set_validate(Tlrn, validate = 0.5)
n.epochs <- 2
Tlrn$callbacks <- mlr3torch::t_clbk("history")
Tlrn$param_set$values$batch_size <- 10
Tlrn$param_set$values$epochs <- n.epochs
Tlrn$param_set$values[c("measures_train","measures_valid")] <- mlr3::msrs(c("regr.rmse"))
Alrn <- mlr3tuning::auto_tuner(
learner = Tlrn,
tuner = mlr3tuning::tnr("internal"),
resampling = mlr3::rsmp("insample"),
measure = mlr3::msr("internal_valid_score", minimize = TRUE),
term_evals = 1,
id="torch_linear",
store_models = TRUE)
Tlrn$validate
Alrn$train(mtask)and I got the following output:
> Tlrn$validate
[1] 0.5
> Alrn$train(mtask)
INFO [13:05:18.134] [bbotk] Starting to optimize 0 parameter(s) with '<TunerBatchInternal>' and '<TerminatorEvals> [n_evals=1, k=0]'
INFO [13:05:18.139] [bbotk] Evaluating 1 configuration(s)
INFO [13:05:18.154] [mlr3] Running benchmark with 1 resampling iterations
INFO [13:05:18.182] [mlr3] Applying learner 'regr.mlp' on task 'mtcars' (iter 1/1)
INFO [13:05:18.394] [mlr3] Finished benchmark
INFO [13:05:18.485] [bbotk] Result of batch 1:
INFO [13:05:18.492] [bbotk] internal_valid_score warnings errors runtime_learners
INFO [13:05:18.492] [bbotk] 101.7036 0 0 0.17
INFO [13:05:18.492] [bbotk] uhash
INFO [13:05:18.492] [bbotk] a211035f-7e87-44aa-8f78-1d35dbbddc29
INFO [13:05:18.518] [bbotk] Finished optimizing after 1 evaluation(s)
INFO [13:05:18.522] [bbotk] Result:
INFO [13:05:18.528] [bbotk] learner_param_vals x_domain internal_valid_score
INFO [13:05:18.528] [bbotk] <list> <list> <num>
INFO [13:05:18.528] [bbotk] <list[18]> <list[0]> 101.7036
Error in learner_torch_train(self, private, super, task, param_vals) :
Learner 'regr.mlp' has measures_valid set, but its validate field is NULL`which is confusing, because in fact the validate field is not NULL (it is 0.5).
I would suggest changing the error message to something like "paradox::to_tune missing in param_set, please add that when using auto_tuner" which is the fix:
Tlrn$param_set$values$epochs <- paradox::to_tune(upper = n.epochs, internal = TRUE)After making that change I get:
> Alrn$train(mtask)
Error in converter(token, private$.sets[[set_index]]$values) :
Assertion on 'patience parameter for LearnerTorch' failed: Must be TRUE.This error message is also confusing, because patience parameter must be integer (not logical/TRUE).
> Tlrn$param_set$values$patience <- TRUE
Error in self$assert(xs, sanitize = TRUE) :
Assertion on 'xs' failed: patience: Must be of type 'single integerish value', not 'logical'.The solution is to set patience to an integer, and set epochs to_tune:
mtask <- mlr3::tsk("mtcars")
Tlrn <- mlr3torch::LearnerTorchMLP$new(task_type="regr")
mlr3::set_validate(Tlrn, validate = 0.5)
n.epochs <- 2
Tlrn$callbacks <- mlr3torch::t_clbk("history")
Tlrn$param_set$values$patience <- n.epochs
Tlrn$param_set$values$batch_size <- 10
Tlrn$param_set$values$epochs <- paradox::to_tune(upper = n.epochs, internal = TRUE)
Tlrn$param_set$values[c("measures_train","measures_valid")] <- mlr3::msrs(c("regr.rmse"))
Alrn <- mlr3tuning::auto_tuner(
learner = Tlrn,
tuner = mlr3tuning::tnr("internal"),
resampling = mlr3::rsmp("insample"),
measure = mlr3::msr("internal_valid_score", minimize = TRUE),
term_evals = 1,
id="torch_linear",
store_models = TRUE)
Tlrn$validate
Alrn$train(mtask)which gives the output:
> Alrn$train(mtask)
INFO [13:09:36.535] [bbotk] Starting to optimize 0 parameter(s) with '<TunerBatchInternal>' and '<TerminatorEvals> [n_evals=1, k=0]'
INFO [13:09:36.540] [bbotk] Evaluating 1 configuration(s)
INFO [13:09:36.552] [mlr3] Running benchmark with 1 resampling iterations
INFO [13:09:36.576] [mlr3] Applying learner 'regr.mlp' on task 'mtcars' (iter 1/1)
INFO [13:09:36.729] [mlr3] Finished benchmark
INFO [13:09:36.822] [bbotk] Result of batch 1:
INFO [13:09:36.828] [bbotk] internal_valid_score warnings errors runtime_learners internal_tuned_values
INFO [13:09:36.828] [bbotk] 59.50827 0 0 0.11 epochs=2
INFO [13:09:36.828] [bbotk] uhash
INFO [13:09:36.828] [bbotk] 762ef3d1-2ce4-4520-bd43-19b30144a1cb
INFO [13:09:36.886] [bbotk] Finished optimizing after 1 evaluation(s)
INFO [13:09:36.891] [bbotk] Result:
INFO [13:09:36.897] [bbotk] internal_tuned_values learner_param_vals x_domain internal_valid_score
INFO [13:09:36.897] [bbotk] <list> <list> <list> <num>
INFO [13:09:36.897] [bbotk] epochs=2 <list[18]> <list[0]> 59.50827Can those two error messages be updated/clarified please?