这是indexloc提供的服务,不要输入任何密码
Skip to content

Conversation

@mmerickel
Copy link
Contributor

avoid copying the entire git repo to a tmp folder when building the app

@mmerickel
Copy link
Contributor Author

This helps but doing a git clone at all is pretty rough on performance for larger repositories. It ends up copying parts of the .git folder as well as a full checkout. Ideally only the current commit could be copied via a worktree, or even just git archive --format=tar master | tar -C /tmp xf - to scrap the git repo entirely but unfortunately I don't think that's a viable option as subsequent commands rely on it being a valid repo in order to grab submodules.

@josegonzalez
Copy link
Member

https://github.com/Kentzo/git-archive-all

Maybe useful?

@mmerickel
Copy link
Contributor Author

Yeah after I looked to see what other things were happening that I wasn't hitting this PR won't work particularly well. For example the --depth=1 assumes your $REV will be in that range. Need a way to archive a particular commit without cloning the entire history of the repo into a temp folder per-build.

@mmerickel
Copy link
Contributor Author

A git worktree add would work nicely but I don't think we are allow to depend on git 2.x.

@josegonzalez
Copy link
Member

We can depend on whatever we want if it provides improved "something" imo. We do that for nginx for instance.

@mmerickel mmerickel force-pushed the optimize-build-clone branch from 11f57b1 to 00563a2 Compare December 15, 2015 22:59
@mmerickel
Copy link
Contributor Author

I've pushed an update that seems to work well and dramatically speeds up my builds on larger repositories.

@josegonzalez
Copy link
Member

Thats pretty clever.

@mmerickel
Copy link
Contributor Author

I'm not super familiar with git submodules but I think this approach may still clone the entire submodule. Using git-archive-all might fix that but is unrelated to my use-case at least. It appears there is a --depth option to git submodule if we wanted to go deeper down the rabbit hole.

@michaelshobbs
Copy link
Member

Does this hamstring us from doing rollbacks later?

@mmerickel
Copy link
Contributor Author

I'm not sure what you mean but afaict these are tmp folders so they will be gone after reboot at the least so if any plugins are relying on them sticking around they're gonna have a bad time. The code is already doing a rm -rf .git on the line below my changes as well so I can't see how it'd cause any extra git-related issues with future features.

@josegonzalez
Copy link
Member

👍 on what @mmerickel said. I'm not sure how this would hurt anything, so long as the state of the original git repo stays as is.

@michaelshobbs
Copy link
Member

My first glance was cursory at best. Looking more closely, this just puts the one ref into the container, yes?

@josegonzalez
Copy link
Member

We could delete the ref inside that new copy of the branch, no?

@mmerickel
Copy link
Contributor Author

Are you referring to the temporary tag as the ref being injected? I could add a line to delete the ref, but again the .git folder is deleted just below this regardless.

@michaelshobbs
Copy link
Member

I think this is fine as is. I was merely confirming what I was reading.

@michaelshobbs
Copy link
Member

Tests should pass now. Please rebase.

@mmerickel mmerickel force-pushed the optimize-build-clone branch 3 times, most recently from c3ea491 to e2e5536 Compare December 16, 2015 21:42
It's not possible to checkout a specific commit into a new repository
using ``git fetch`` and so a temporary tag is added to the host
repository, then fetched.
@mmerickel mmerickel force-pushed the optimize-build-clone branch from e2e5536 to d31ac73 Compare December 16, 2015 22:23
@mmerickel
Copy link
Contributor Author

Things should be good after these tests finish running.

@u2mejc
Copy link
Contributor

u2mejc commented Dec 16, 2015

Gave it the old college test locally. lgtm 👍

michaelshobbs added a commit that referenced this pull request Dec 16, 2015
optimize git clone for building using depth=1
@michaelshobbs michaelshobbs merged commit b3e9893 into dokku:master Dec 16, 2015
@josegonzalez josegonzalez changed the title optimize git clone for building using depth=1 Optimize git clone for large repositories Dec 23, 2015
@josegonzalez
Copy link
Member

This seems to fail for repositories where submodules are not at HEAD due to the added --depth=1 flag. As that is a breaking change, I am thinking we should either back it out or allow users to fallback to non-depth=1. Thoughts @mmerickel ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants