Validation of meta.yaml in cross-org repos #3680
Conversation
Repos, in particular cross-org ones, may have their own JSON rules
… is the component name
|
I can see some CI failures when the YAML schema can't be found. Would I check whether |
There was a problem hiding this comment.
Could we add a test where we try to lint a module from the nf-core-test repository https://github.com/nf-core-test/modules.git? Thanks!
| @@ -72,7 +72,7 @@ def meta_yml(module_lint_object: ComponentLint, module: NFCoreComponent, allow_m | |||
| # Confirm that the meta.yml file is valid according to the JSON schema | |||
| valid_meta_yml = False | |||
| try: | |||
| with open(Path(module_lint_object.modules_repo.local_repo_dir, "modules/meta-schema.json")) as fh: | |||
| with open(Path(module.base_dir, "modules/meta-schema.json")) as fh: | |||
There was a problem hiding this comment.
I am not sure we should change this. module_lint_object.modules_repo.local_repo_dir is checking the cloned modules repository, which should be the one that is being used.
I think what should be changed is the repo that we are cloning to the cache. I was trying to make some test with the test modules repo but there seems to be an error, my suggestion would be to fix that repo until it is useable and then we can figure out the best way to adapt it.
|
Hi again, I could do some tests, the issue is that you have to provide the git remote in the linting command, like |
| @@ -115,7 +115,7 @@ def meta_yml(subworkflow_lint_object, subworkflow, allow_missing: bool = False): | |||
| # join included modules and included subworkflows in a single list | |||
| included_components_names = [component["name"] for component in included_components] | |||
| if "components" in meta_yaml: | |||
| meta_components = [x for x in meta_yaml["components"]] | |||
| meta_components = [x if isinstance(x, str) else list(x)[0] for x in meta_yaml["components"]] | |||
I can confirm it works for subworkflows (I still need to check modules). But I'm still wondering: why is nf-core making a git clone of the remote of the current repository instead of simply using the existing checkout that the user is trying to lint ? |
We're making our way towards setting up our cross-organisation repository for modules and sub-workflows.
Linting is now failing and here is what I propose to change:
git_remotekey under component names: https://nf-co.re/docs/tutorials/external_usage/cross_organisational_subworkflowsThe problem is that
nf-core lintuses~/.config/nfcore/nf-core/modules/subworkflows/yaml-schema.jsonwhich is not aware of the possibility of that extra key. I therefore propose to use the JSON that is shipped in the user repository itselfsubworkflow.base_dir. For consistency, I make modules linting usemodule.base_dir.This change in fact gives modules repo owners the liberty of defining their own yaml validation rules.
meta_yaml["components"]can now contain dictionaries, it needs a bit of parsing to get the component namesBest,
Matthieu
PR checklist
CHANGELOG.mdis updateddocsis updated