-
Notifications
You must be signed in to change notification settings - Fork 214
Open
Labels
downloadnf-core downloadnf-core download
Description
Note: This was commented on main the downloads functionality pull request #3634
To comment on #3634 (comment) and #3634 (comment) , here are my observations from testing combinations of cache/library parameters and options.
- The proposed implementation/purpose of
-u copy
is to create a standalone copy of the pipeline. Thus, its localsingularity-images/
directory must be complete, andnextflow.config
must be modified to point at it. The pipeline can then run without a cache/library directory externally set.
Rightly, the command takes images from the pre-existing cache or library if possible, rather than downloading absolutely everything.
The current implementation also copies images that are downloaded to the cache directory. This has no runtime effect but honours the purpose of the cache/library: after the command is run, they together hold all images as well, thus helping future / other invocations of Nextflow. Note that the cache may not hold all images itself, as some may rather be in the library only. - The proposed implementation/purpose of
-u amend
is to create a copy of the pipeline that entirely relies on the external cache alone. That's why the localsingularity-images/
directory is left empty and all images are rather deposited into the cache.nextflow.config
is not modified.
Here also, the command only downloads the images that are not present in the cache/library.
Contrary to-u copy
, images are copied to the cache to make it complete on its own, rather than allowing some images to be in the library only. - However I don't understand the implementation/purpose of skipping the
-u
option. When I runnf-core pipelines download
without-u
, the localsingularity-images/
directory is made complete butnextflow.config
isn't updated, meaningsingularity-images/
is not used. Fortunately, like-u copy
, the cache is updated, so the pipeline can still run offline provided the cache (and the library) are set externally. It's essentially something in between-u copy
and-u amend
, but I can't define the rationale.
My thinking is that:
- In
-u amend
mode, no need to duplicate images from the library to the cache. In other words, assume that the pipeline will run with both the cache and library set the same way as innf-core pipelines download
, rather than only the cache. - In
-u amend
mode, no need to create a localsingularity-images/
directory if it's left empty. - Don't allow
-u
to be unset ?
Originally posted by @muffato in #3634 (comment)
Metadata
Metadata
Assignees
Labels
downloadnf-core downloadnf-core download
Type
Projects
Status
Todo