Update conda env sorting #3591

edmundmiller · 2025-05-30T02:37:41Z

Adds extra tests. Didn't realize #3584 added the environment.yml listing so recently.

This fixes an issue with the Seqera containers' naming.

The Seqera containers site doesn't care about the context of what module the
image is for. It just sorts the dependencies.

This avoids having multiple containers with a bunch of different names.

PR checklist

This comment contains a description of changes (with reason)
CHANGELOG.md is updated
If you've fixed a bug or added code that should be tested, add tests!
Documentation in docs is updated

mashehu · 2025-05-30T10:43:59Z

Just had a quick look on the phone. for me having first pip and then immediately followed by pip dependencies made more sense. Is there a standard here?

mashehu · 2025-05-30T11:12:32Z

@nf-core-bot fix linting

mashehu

Please use ruaml for yaml parsing so we don't lose comments.

Also the channel order has a meaning, so I think we shouldn't touch it.

nf_core/modules/lint/environment_yml.py

edmundmiller · 2025-05-30T15:36:26Z

Please use ruaml for yaml parsing so we don't lose comments.

I was just following the previous standard 🙃 I had it originally implemented with ruaml.

edmundmiller · 2025-05-30T17:58:43Z

@nf-core-bot fix linting

mashehu

I am confused. what's the difference between the functionality in this PR vs #3584
Don't the create the same result in the end?
The additional tests are fine.

Wanted to just comment not yet remove

mashehu · 2025-05-30T19:22:50Z

*approve

Stupid github phone app...

edmundmiller · 2025-05-30T19:31:27Z

No extra functionality, I just missed that you had fixed this 3 days ago 🙃 .

codecov · 2025-06-03T14:36:32Z

Codecov Report

Attention: Patch coverage is 98.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 76.90%. Comparing base (619dd6e) to head (ede037b).

Files with missing lines	Patch %	Lines
nf_core/modules/lint/environment_yml.py	98.00%	1 Missing ⚠️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

nf_core/modules/lint/environment_yml.py

…dling - Renamed test functions for clarity. - Introduced DummyModule and DummyLint classes to simulate module and lint behavior. - Enhanced tests for handling invalid, empty, and missing dependencies in environment.yml files. - Improved assertions to validate sorting and schema compliance.

- Replaced yaml library with ruamel.yaml for improved YAML processing. - Added schema validation for environment.yml files. - Implemented sorting for dependencies and channels, ensuring proper order. - Updated file writing to include schema lines and sorted content. - Enhanced logging for sorting actions and validation results.

…processing - Updated the YAML library from ruamel.yaml to PyYAML for improved compatibility. - Changed YAML loading to use safe_load for better security. - Enhanced YAML dumping with specific formatting options for clarity. - Adjusted error handling to reflect the new library usage.

…ent.yml processing" This reverts commit 34d5b00. Co-authored-by: mashehu <mashehu@users.noreply.github.com>

…endencies Co-authored-by: mashehu <mashehu3@gmail.com>

…l sorting - Removed channel sorting logic from environment.yml processing. - Updated related tests to reflect channel preservation instead of sorting. - Adjusted logging messages to focus solely on dependency sorting. Co-authored-by: mashehu <mashehu3@gmail.com>

- Introduced factory fixtures for creating DummyModule and DummyLint instances to streamline test setup. - Updated tests to utilize the new setup_lint_environment fixture for improved clarity and maintainability. - Enhanced assertions for YAML parsing results in the environment.yml sorting tests. - Added parameterized tests for handling invalid and empty YAML files.

- Updated pytest.ini_options in pyproject.toml to include a new "integration" marker for better test categorization.

…nt_test attribute - Changed the assertion to check for the "environment_yml_sorted" in the lint.passed list using the lint_test attribute instead of the previous tuple structure.

but it is a link to another json file

mashehu · 2025-07-14T14:49:11Z

because we already have a working and tested sorting, how difficult would it be to revert the changes to it in this PR and only keep the tests? or is there a bigger difference between the two implementations i don't see?

edmundmiller · 2025-07-15T13:46:41Z

because we already have a working and tested sorting, how difficult would it be to revert the changes to it in this PR and only keep the tests? or is there a bigger difference between the two implementations i don't see?

It uses ruaml which was a request you had

Please use ruaml for yaml parsing so we don't lose comments.

If you'd feel more comfortable, we can break the tests into a PR and then the changes into a second PR.

mashehu · 2025-07-15T16:15:20Z

It uses ruaml which was a request you had

That was before I understood that it re-implements the sorting functionality 🙂

edmundmiller changed the title ~~Add Conda env sorting~~ Update conda env sorting May 30, 2025

edmundmiller self-assigned this May 30, 2025

edmundmiller added this to Seqera Containers and nf-core infrastructure projects May 30, 2025

edmundmiller added this to the 3.4.0 milestone May 30, 2025

edmundmiller linked an issue May 30, 2025 that may be closed by this pull request

Sort modules conda environment files #3590

Closed

mashehu requested changes May 30, 2025

View reviewed changes

mashehu reviewed May 30, 2025

View reviewed changes

nf_core/modules/lint/environment_yml.py Outdated Show resolved Hide resolved

edmundmiller mentioned this pull request May 30, 2025

Refactor module lint tests #3592

Draft

4 tasks

edmundmiller force-pushed the env-sorting branch from 5bb9075 to 78cd7ee Compare May 30, 2025 17:57

edmundmiller requested a review from mashehu May 30, 2025 17:58

edmundmiller moved this to In progress in Seqera Containers May 30, 2025

edmundmiller moved this to In Progress in nf-core infrastructure projects May 30, 2025

edmundmiller force-pushed the env-sorting branch from 7b2e9e7 to 5e47ac1 Compare May 30, 2025 18:10

mashehu previously approved these changes May 30, 2025

View reviewed changes

edmundmiller requested a review from mashehu June 3, 2025 14:49

edmundmiller enabled auto-merge June 3, 2025 14:49

mirpedrol reviewed Jun 4, 2025

View reviewed changes

nf_core/modules/lint/environment_yml.py Outdated Show resolved Hide resolved

nf_core/modules/lint/environment_yml.py Show resolved Hide resolved

mashehu added the awaiting-changes Needs changes after a review. Remove label when ready to be reviewed again. label Jun 16, 2025

edmundmiller requested a review from mirpedrol July 1, 2025 14:16

edmundmiller added 2 commits July 1, 2025 09:45

test(#3590): Copy tests over from env sorter

cdda107

edmundmiller and others added 13 commits July 1, 2025 09:45

test(#3590): Move integration tests to new lint test file

c62ccb9

test(#3590): Update dependency order on a test

feabc68

Revert "refactor(#3590): Replace ruamel.yaml with PyYAML for environm…

e4026f3

…ent.yml processing" This reverts commit 34d5b00. Co-authored-by: mashehu <mashehu@users.noreply.github.com>

fix(#3590): Having first pip and then immediately followed by pip dep…

d8eff3c

…endencies Co-authored-by: mashehu <mashehu3@gmail.com>

style: Clean up parameterized tests

8c4839c

style: Fix code linting

c758daf

chore: Add integration marker to pytest configuration

c901b4c

- Updated pytest.ini_options in pyproject.toml to include a new "integration" marker for better test categorization.

fix(test): Update assertion in environment.yml sorting test to use li…

4b8226d

…nt_test attribute - Changed the assertion to check for the "environment_yml_sorted" in the lint.passed list using the lint_test attribute instead of the previous tuple structure.

fix(modules): Check if they are not the exact same lines

ede037b

but it is a link to another json file

edmundmiller force-pushed the env-sorting branch from 01541c8 to ede037b Compare July 1, 2025 14:45

mashehu modified the milestones: 3.4.0, 3.5.0 Jul 8, 2025

edmundmiller removed the awaiting-changes Needs changes after a review. Remove label when ready to be reviewed again. label Jul 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update conda env sorting #3591

Update conda env sorting #3591

Uh oh!

edmundmiller commented May 30, 2025 •

edited

Loading

Uh oh!

mashehu commented May 30, 2025

Uh oh!

mashehu commented May 30, 2025

Uh oh!

mashehu left a comment

Uh oh!

Uh oh!

edmundmiller commented May 30, 2025

Uh oh!

edmundmiller commented May 30, 2025

Uh oh!

mashehu left a comment

Uh oh!

mashehu commented May 30, 2025

Uh oh!

edmundmiller commented May 30, 2025

Uh oh!

codecov bot commented Jun 3, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

mashehu commented Jul 14, 2025

Uh oh!

edmundmiller commented Jul 15, 2025

Uh oh!

mashehu commented Jul 15, 2025

Uh oh!

Uh oh!

Update conda env sorting #3591

Are you sure you want to change the base?

Update conda env sorting #3591

Uh oh!

Conversation

edmundmiller commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR checklist

Uh oh!

mashehu commented May 30, 2025

Uh oh!

mashehu commented May 30, 2025

Uh oh!

mashehu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

edmundmiller commented May 30, 2025

Uh oh!

edmundmiller commented May 30, 2025

Uh oh!

mashehu left a comment

Choose a reason for hiding this comment

Uh oh!

mashehu commented May 30, 2025

Uh oh!

edmundmiller commented May 30, 2025

Uh oh!

codecov bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

mashehu commented Jul 14, 2025

Uh oh!

edmundmiller commented Jul 15, 2025

Uh oh!

mashehu commented Jul 15, 2025

Uh oh!

Uh oh!

edmundmiller commented May 30, 2025 •

edited

Loading

codecov bot commented Jun 3, 2025 •

edited

Loading