这是indexloc提供的服务，不要输入任何密码
Skip to content

Support HMAC signing on artifact uploads #892

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 30 commits into from
Mar 25, 2022
Merged

Support HMAC signing on artifact uploads #892

merged 30 commits into from
Mar 25, 2022

Conversation

gaspar09
Copy link
Contributor

@gaspar09 gaspar09 commented Mar 16, 2022 •
edited
Loading

Introduce RemoteCacheOptions field to turbo.json
Adds initial test scaffolding for turbo json config parsing.
Implement HMAC Signature sent on uploads to the remote cache via x-artifact-tag header.
Implement Signature Stream Validator on artifact downloads.
Download full response body and verify before untar when the signature/verification is enabled.

In followup:

  • Restrict uploads to teamId specified in remoteCacheOptions

gaspar09
gaspar09 commented Mar 16, 2022
View reviewed changes
jaredpalmer
jaredpalmer reviewed Mar 16, 2022
View reviewed changes
@vercel
Copy link

vercel bot commented Mar 16, 2022 •
edited
Loading

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/vercel/turbo-site/4CexmiMV8Wg6qZgmaAZxjZsx39Vr
✅ Preview: https://turbo-site-git-gaspar-remote-cache-options.vercel.sh

@jaredpalmer
Copy link
Contributor

annoying nit: we also need to update schema.d.ts w/ comments for our JSONSchema to play nice with VSCode/IDE automcomplete here: https://github.com/vercel/turborepo/blob/main/docs/schema.d.ts

There is already a PR by @Schniz to move it all to be Go, but Go isn't installed in build container on Vercel.

@gaspar09
Explicitly read the artifact into memory before uploading to the remo…
7386176 
…te cache api

Read the tar artifact into memory so that we can compute the content-md5
This read was already happening implicitly via the retryableHttp module.
Add content-md5 to header

Revert "Revert trailer md5 (#916)"
This reverts commit 862e24f.
@gaspar09 gaspar09 changed the title Introduce RemoteCacheOptions field to turbo.json Support HMAC signing on artifact uploads Mar 22, 2022
@gaspar09
Implement artifact HMAC signing as ArtifactSignatureAuthentication
bb2b646 
The hmac tag is sent on uploads to the remote cache via x-artifact-tag header.

TODO:
- Compute artifact signature from streaming artifact download
- Validate x-aritfact-tag header once download completes
@gaspar09 gaspar09 removed the pr: automerge Kodiak will merge these automatically after checks pass label Mar 24, 2022
@gaspar09 gaspar09 requested a review from gsoltis March 24, 2022 23:07
jaredpalmer
jaredpalmer suggested changes Mar 25, 2022
View reviewed changes
cli/internal/fs/testdata/turbo.json Outdated
"cache": false
}
},
"remoteCacheOptions": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: aren't Options implied? Should it just be remoteCache

cli/internal/fs/testdata/turbo.json Outdated
},
"remoteCacheOptions": {
"teamId": "team_id",
"signatureOptions": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we just call this signature?

@jaredpalmer
Copy link
Contributor

jaredpalmer commented Mar 25, 2022 •
edited
Loading

In another follow up, we should allow folks to set cacheDirectory and once #833 remoteOnly. Wondering if we want to refactor the payload to cache

@gaspar09 gaspar09 mentioned this pull request Mar 25, 2022
Closed
@gaspar09
Copy link
Contributor Author

In another follow up, we should allow folks to set cacheDirectory and once #833 remoteOnly. Wondering if we want to refactor the payload to cache

noted:
#952

@gaspar09 gaspar09 requested a review from jaredpalmer March 25, 2022 20:43
@kodiakhq kodiakhq bot merged commit dbc94b5 into main Mar 25, 2022
@kodiakhq kodiakhq bot deleted the gaspar/remote-cache-options branch March 25, 2022 20:57
@jaredpalmer
Copy link
Contributor

Lets get started on docs for this

@weyert weyert mentioned this pull request Mar 28, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jaredpalmer jaredpalmer jaredpalmer approved these changes

@gsoltis gsoltis Awaiting requested review from gsoltis

Assignees

@gaspar09 gaspar09

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@gaspar09 @jaredpalmer @gsoltis