feat: Add prune support for Yarn 3 #2019

chris-olszewski · 2022-09-20T17:19:20Z

Add prune support for Yarn 3. In order to support this we also support marshalling to the yarn lockfile data format. Due to the strict nature of yarn install --immutable BerryLockfile has internal structures that aren't directly serialized to the generated lockfile. Future work will be required to make the Subgraph method correctly prune these internal structures.

A quick rundown of the structure of a yarn lockfile:

Descriptors are packages with a version range used to index and find the resolved entry for a package. Resolved entries have multiple descriptors if workspace packages or transitive dependencies use the same package with overlapping semver ranges. All descriptors are joined with , to create the key to the lockfile entry
Locators are the resolved version of a descriptor where instead of a version range there is a concrete version. Each locator maps directly to an entry in the lockfile and is found in the resolution field.

Known issues:

This doesn't prune patches produced by yarn patch as this requires pruning of package.json which we don't support yet.
Yarn bundles some dependencies that default plugins require. It appears that if your lockfile contains an entry for these dependencies, Yarn expects that you should include the descriptor that is used in the bundled version. To see the bundled dependencies search for ^7.8.3 in this file. Some of these are very common packages so I'm afraid that this might be a case that's easy to hit. Looking into if this is desired behavior for Yarn and if there's any way to figure this out without parsing the minified yarn JS.

vercel · 2022-09-20T17:19:25Z

@chris-olszewski is attempting to deploy a commit to the Vercel Team on Vercel.

A member of the Team first needs to authorize it.

quinnturner · 2022-09-20T21:40:40Z

cli/internal/lockfile/berry_lockfile.go

+	LinkType   string `yaml:"linkType,omitempty"`
+	Resolution string `yaml:"resolution,omitempty"`
+	Checksum   string `yaml:"checksum,omitempty"`
+	Conditions string `yaml:"conditions,omitempty"`


Hi! If you're using the same YAML parser that existed before, you may need to order these based on their presence in the lockfile definition.

This order follows the order present in the Yarn Berry lockfile definition:

type BerryLockfileEntry struct { // resolved version for the particular entry based on the provided semver revision Version string `yaml:"version"` CacheKey string `yaml:"cacheKey,omitempty"` // only present in the __metadata object Resolution string `yaml:"resolution,omitempty"` // the list of unresolved modules and revisions (e.g. type-detect : ^4.0.0) Dependencies map[string]string `yaml:"dependencies,omitempty"` // the list of unresolved modules and revisions (e.g. type-detect : ^4.0.0) PeerDependencies map[string]string `yaml:"peerDependencies,omitempty"` PeerDependenciesMeta map[string]map[string]bool `yaml:"peerDependenciesMeta,omitempty"` Bin map[string]string `yaml:"bin,omitempty"` OptionalDependencies map[string]string `yaml:"optionalDependencies,omitempty"` Checksum string `yaml:"checksum,omitempty"` Conditions string `yaml:"conditions,omitempty"` LanguageName string `yaml:"languageName,omitempty"` LinkType string `yaml:"linkType,omitempty"` }

The current version is missing OptionalDependencies. Is that intentional?

If the order doesn't matter due to the implementation approach, you can ignore this if you'd like!

I'm just using yaml to parse the lockfile. Sadly due to some oddities with the Yarn lockfile format I had to port the serialization logic to Go instead of using the yaml library.

Note that currently patches aren't properly supported when decoding and will be unable to be found unless the caller has the proper params added to the patch semver

… entries

vercel · 2022-09-21T18:47:58Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
turbo-site	✅ Ready (Inspect)	Visit Preview	Sep 21, 2022 at 7:23PM (UTC)

gsoltis · 2022-09-21T20:12:10Z

Force-merging, error looks unrelated.

### Description Adds a new Berry lockfile implementation that correctly handles [resolutions](https://yarnpkg.com/configuration/manifest/#resolutions). See the description of #2019 for an overview of the berry lockfile format. Note for reviewers: - I suggest reviewing `identifiers.rs` first as `Ident`, `Descriptor`, and `Locator` are the building blocks for most of the lockfile logic - We cannot use `serde_yaml` for serialization as berry expects some strings to be quoted that are not necessarily required to be quoted by the YAML spec. - The underlying lockfile data is passed into the struct that contains the lockfile logic, this allows us to borrow from the data when building up the internal data structures. Down the road this could be changed to use `Pin` if this proves to be painful from an ergonomics perspective ### Testing Instructions Ported existing unit tests and added new ones for the resolutions logic. The PR that switches this Rust impl on will include instructions on how to do e2e testing. --------- Co-authored-by: Alexander Lyon <arlyon@me.com>

quinnturner reviewed Sep 20, 2022

View reviewed changes

chris-olszewski marked this pull request as ready for review September 20, 2022 22:53

chris-olszewski requested a review from a team as a code owner September 20, 2022 22:53

chris-olszewski requested review from gsoltis and tknickman September 20, 2022 22:53

chris-olszewski added 7 commits September 20, 2022 15:54

Proper support for decoding lockfile entries

b814de2

Note that currently patches aren't properly supported when decoding and will be unable to be found unless the caller has the proper params added to the patch semver

add encoding for berry lockfile

84f2457

support builtin patches

a653f36

corrently pruned entries, but extra descriptors

8a7d474

Add failing test for descriptor pruning

44644a4

prune descriptors

d13edef

Add berry to e2e prune tests

73017d8

chris-olszewski force-pushed the add_berry_prune_impl branch from deab32a to 73017d8 Compare September 20, 2022 22:55

include descriptors in pruned lockfile that come from package extensions

1ecd6de

gsoltis approved these changes Sep 21, 2022

View reviewed changes

Greg Soltis and others added 3 commits September 21, 2022 09:55

Merge branch 'main' into add_berry_prune_impl

40a2e26

handle crlf newlines in attempt to appease e2e tests

b89fc46

fix: convert workspace paths to unix paths before looking up lockfile…

5fbf7ab

… entries

chris-olszewski added the pr: automerge Kodiak will merge these automatically after checks pass label Sep 21, 2022

vercel bot deployed to Preview September 21, 2022 18:48 View deployment

Greg Soltis added 2 commits September 21, 2022 12:05

Merge branch 'main' into add_berry_prune_impl

93ffd80

Merge branch 'main' into add_berry_prune_impl

ed4cd8c

vercel bot deployed to Preview September 21, 2022 19:23 View deployment

gsoltis merged commit 39e19bb into vercel:main Sep 21, 2022

This was referenced Sep 21, 2022

Prune Command Writes Old Metadata Version to Yarn Lockfile #1498

Closed

Yarn after turbo prune -> Syntax Error: Expected "\n", "\r", or "\r\n" but ":" found. #1975

Closed

Empty resolutions after turbo prune #1842

Closed

nathanhammond mentioned this pull request Sep 22, 2022

Support yarn 3 lockfile entries #643

Closed

chris-olszewski mentioned this pull request Apr 14, 2023

feat: Add rust implementation of Yarn3+ lockfile #4589

Merged

krinoid mentioned this pull request May 15, 2023

[turborepo] turbo prune throws unable to parse yaml: data did not match any variant of untagged enum StringOrNum #4925

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add prune support for Yarn 3 #2019

feat: Add prune support for Yarn 3 #2019

Uh oh!

chris-olszewski commented Sep 20, 2022 •

edited

Loading

Uh oh!

vercel bot commented Sep 20, 2022

Uh oh!

quinnturner Sep 20, 2022 •

edited

Loading

Uh oh!

chris-olszewski Sep 20, 2022

Uh oh!

vercel bot commented Sep 21, 2022 •

edited

Loading

Uh oh!

gsoltis commented Sep 21, 2022

Uh oh!

Uh oh!

feat: Add prune support for Yarn 3 #2019

feat: Add prune support for Yarn 3 #2019

Uh oh!

Conversation

chris-olszewski commented Sep 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel bot commented Sep 20, 2022

Uh oh!

quinnturner Sep 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chris-olszewski Sep 20, 2022

Choose a reason for hiding this comment

Uh oh!

vercel bot commented Sep 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gsoltis commented Sep 21, 2022

Uh oh!

Uh oh!

chris-olszewski commented Sep 20, 2022 •

edited

Loading

quinnturner Sep 20, 2022 •

edited

Loading

vercel bot commented Sep 21, 2022 •

edited

Loading