+
Skip to content

Pipeline download purpose of cache-utilization parameters after refactoring #3731

@JulianFlesch

Description

@JulianFlesch

Note: This was commented on main the downloads functionality pull request #3634

To comment on #3634 (comment) and #3634 (comment) , here are my observations from testing combinations of cache/library parameters and options.

  1. The proposed implementation/purpose of -u copy is to create a standalone copy of the pipeline. Thus, its local singularity-images/ directory must be complete, and nextflow.config must be modified to point at it. The pipeline can then run without a cache/library directory externally set.
    Rightly, the command takes images from the pre-existing cache or library if possible, rather than downloading absolutely everything.
    The current implementation also copies images that are downloaded to the cache directory. This has no runtime effect but honours the purpose of the cache/library: after the command is run, they together hold all images as well, thus helping future / other invocations of Nextflow. Note that the cache may not hold all images itself, as some may rather be in the library only.
  2. The proposed implementation/purpose of -u amend is to create a copy of the pipeline that entirely relies on the external cache alone. That's why the local singularity-images/ directory is left empty and all images are rather deposited into the cache. nextflow.config is not modified.
    Here also, the command only downloads the images that are not present in the cache/library.
    Contrary to -u copy, images are copied to the cache to make it complete on its own, rather than allowing some images to be in the library only.
  3. However I don't understand the implementation/purpose of skipping the -u option. When I run nf-core pipelines download without -u, the local singularity-images/ directory is made complete but nextflow.config isn't updated, meaning singularity-images/ is not used. Fortunately, like -u copy, the cache is updated, so the pipeline can still run offline provided the cache (and the library) are set externally. It's essentially something in between -u copy and -u amend, but I can't define the rationale.

My thinking is that:

  1. In -u amend mode, no need to duplicate images from the library to the cache. In other words, assume that the pipeline will run with both the cache and library set the same way as in nf-core pipelines download, rather than only the cache.
  2. In -u amend mode, no need to create a local singularity-images/ directory if it's left empty.
  3. Don't allow -u to be unset ?

Originally posted by @muffato in #3634 (comment)

Metadata

Metadata

Assignees

Labels

downloadnf-core download

Type

No type

Projects

Status

Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    点击 这是indexloc提供的php浏览器服务,不要输入任何密码和下载