+
Skip to content

Conversation

muffato
Copy link
Member

@muffato muffato commented Jul 15, 2025

We're making our way towards setting up our cross-organisation repository for modules and sub-workflows.

Linting is now failing and here is what I propose to change:

  1. In order to do cross-org repositories, we need to modify the JSON schema to allow the git_remote key under component names: https://nf-co.re/docs/tutorials/external_usage/cross_organisational_subworkflows
    The problem is that nf-core lint uses ~/.config/nfcore/nf-core/modules/subworkflows/yaml-schema.json which is not aware of the possibility of that extra key. I therefore propose to use the JSON that is shipped in the user repository itself subworkflow.base_dir. For consistency, I make modules linting use module.base_dir.
    This change in fact gives modules repo owners the liberty of defining their own yaml validation rules.
  2. Since meta_yaml["components"] can now contain dictionaries, it needs a bit of parsing to get the component names

Best,
Matthieu

PR checklist

  • This comment contains a description of changes (with reason)
  • CHANGELOG.md is updated
  • If you've fixed a bug or added code that should be tested, add tests!
  • Documentation in docs is updated

@muffato
Copy link
Member Author

muffato commented Jul 15, 2025

I can see some CI failures when the YAML schema can't be found. Would I check whether Path(subworkflow.base_dir, "subworkflows/yaml-schema.json") exists and fallback to Path(subworkflow_lint_object.modules_repo.local_repo_dir, "subworkflows/yaml-schema.json") ?

Copy link
Member

@mirpedrol mirpedrol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a test where we try to lint a module from the nf-core-test repository https://github.com/nf-core-test/modules.git? Thanks!

@mirpedrol
Copy link
Member

Hi again, I could do some tests, the issue is that you have to provide the git remote in the linting command, like nf-core subworkflows --git-remote https://github.com/<custom_org>/modules.git lint, that way module_lint_object.modules_repo.local_repo_dir will point to the right path

@muffato
Copy link
Member Author

muffato commented Jul 16, 2025

Hi again, I could do some tests, the issue is that you have to provide the git remote in the linting command, like nf-core subworkflows --git-remote https://github.com/<custom_org>/modules.git lint, that way module_lint_object.modules_repo.local_repo_dir will point to the right path

I can confirm it works for subworkflows (I still need to check modules). But I'm still wondering: why is nf-core making a git clone of the remote of the current repository instead of simply using the existing checkout that the user is trying to lint ?

@mashehu
Copy link
Contributor

mashehu commented Jul 21, 2025

If i understand it correctly, we use this setup to handle multiple remote repositories correctly, so it always compares against the correct one

@muffato muffato force-pushed the crossorg_yaml_validation branch from 6b80db2 to 924721a Compare July 21, 2025 13:01
@muffato muffato force-pushed the crossorg_yaml_validation branch from 924721a to ae7ca23 Compare July 21, 2025 13:02
@muffato
Copy link
Member Author

muffato commented Jul 21, 2025

Alright. I've now removed that part of the change.

Copy link

codecov bot commented Jul 24, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.10%. Comparing base (5081480) to head (ae7ca23).
⚠️ Report is 41 commits behind head on dev.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Member

@mirpedrol mirpedrol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mirpedrol mirpedrol merged commit 9bff2b9 into dev Aug 1, 2025
170 of 171 checks passed
@mirpedrol mirpedrol deleted the crossorg_yaml_validation branch August 1, 2025 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

点击 这是indexloc提供的php浏览器服务,不要输入任何密码和下载