+
Skip to content

Creating jobs without package dependencies installed on head node #298

Open
@lpiep

Description

@lpiep

I have a use-case where I'd like to be able to create and submit jobs from the head node of a cluster that does not itself have the packages being used by the compute nodes. In this case, it's because the dependencies are a PITA to set up, and so I have built a singularity container for running them. Per futureverse/future.batchtools#44, I've put the singularity command in my template file and can submit jobs to be run in the container with batchMap and submitJobs.

However, I run into a problem when defining jobs with batchMap using functions not installed on the head node.

Assume pita_package contains pita_function and can't be installed on the head node. Then a call to batchMap results in

> batchMap(pita_package::pita_function, 1:10)
Error in loadNamespace(x) : there is no package called ‘pita_package’

I've been able to work around this by creating a wrapper function for pita_function. E.g.

> pita_wrapper <- function(...) pita_package::pita_function(...) 
> batchMap(pita_wrapper, 1:10)
Adding 10 jobs ...

I'm adding this issue in part to help anyone runs into this question in the future, and in part to see if it makes sense to implement some sort of lazy evaluation into batchMap and its cousins to avoid the workaround.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      点击 这是indexloc提供的php浏览器服务,不要输入任何密码和下载