这是indexloc提供的服务,不要输入任何密码
Skip to content

Refactor core scheduler #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 21 commits into from
Closed

Refactor core scheduler #20

wants to merge 21 commits into from

Conversation

jaredpalmer
Copy link
Contributor

@jaredpalmer jaredpalmer commented Oct 29, 2021

This completely reimplements the task graph construction and fixes a long standing bug, which resulted in incorrect over-builds (running more tasks than necessary) and worse, incorrect builds, when the user forgot to specify the full set of possible targets.

In addition, this PR refactors Turbo's scheduler into a module which nicely separates concerns:

  • context.go: Creating the topological package graph and preliminary hashing
  • run.go: Determining which packages and targets are in execution scope given the pipeline and CLI arguments
  • scheduler.go: Constructing and running the orthogonal task graph given the topological graph and scope

In practice, the scheduler's API is now abstracted into a builder pattern as follows:

API

func NewScheduler(topologicalGraph *dag.AcyclicGraph) *scheduler
type SchedulerExecutionOptions struct{ ... }
type Task struct{ ... }

Usage

func main() {
	var g dag.AcyclicGraph
	g.Add("a")
	g.Add("b")
	g.Add("c")
	g.Connect(dag.BasicEdge("c", "b"))
	g.Connect(dag.BasicEdge("c", "a"))

	p := NewScheduler(&g)

    // Create a set of topological task deps (like those between packages)
	topoDeps := make(util.Set)
    // Create a set of regular task deps (like those within the same package)
	deps := make(util.Set)

	// Add `build` to the set of topological deps (e.g. `^build`)
    // This translates to: <task> depends on its dependencies' `build` task
	topoDeps.Add("build")

    // Add `prepare` to the set of deps.
    // This translates to: <task> depends on its own `prepare` `task`
	deps.Add("prepare")

    // Add tasks to the scheduler (would be based on some kind of pipeline definition)
	p.AddTask(&Task{
		Name:     "build",
		TopoDeps: topoDeps, // build depends on its deps' build command
		Deps:     deps, // build depends on its own prepare command
		Run: func(taskId string) error {            
			fmt.Println(taskId)
			return nil
		},
	})

	p.AddTask(&Task{
		Name:     "test",
		TopoDeps: topoDeps, // test depends on its deps' build command
		Deps:     deps, // test depends on its own prepare command
		Run: func(taskId string) error {
			fmt.Println(taskId)
			return nil
		},
	})

	p.AddTask(&Task{
		Name: "prepare",
		Run: func(taskId string) error {
			fmt.Println(taskId)
			return nil
		},
	})

    // Execute constructs a new task graph based on current topologic dep graph and execution scope (packages and task names). It then
    // walks the task graph just like before. It will ensure that all transitive tasks are correctly included so that their is no need know the full set of tasks ahead of time
    errs := p.Execute(&SchedulerExecutionOptions{
		Packages:   nil, // optionally restrict packages in scope
		TaskNames:  []string{"test"},  // optionally restrict task's in scope
		Concurreny: 10, // set graph concurrency
		Parallel:   false, // control parallel running (if true, it removes all task graph edges)
	})

	for _, err := range errs {
		log.Fatalf("%v", err)
	}     

	log.Println(strings.TrimSpace(p.TaskGraph.String()))
}     

Output

___ROOT___
a#build
  ___ROOT___
a#test
  ___ROOT___
b#build
  ___ROOT___
b#test
  ___ROOT___
c#test
  a#build
  b#build

__ROOT__ is the internal name for the root node of the graph which is always necessary so that the graph always closes and there are no dangling nodes that are not connected to the rest of the graph

@vercel
Copy link

vercel bot commented Oct 29, 2021

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/vercel/turbo-site/8npLpd2JK4KGwsiM932r5bnpVp6N
✅ Preview: https://turbo-site-git-refactor-to-pipeline.vercel.sh

@jaredpalmer jaredpalmer added this to the Turbo Launch milestone Oct 29, 2021
@jaredpalmer jaredpalmer self-assigned this Oct 29, 2021
@jaredpalmer jaredpalmer changed the title Refactor core into pipeline Refactor core scheduler Nov 1, 2021
@jaredpalmer jaredpalmer changed the title Refactor core scheduler Refactor core scheduler Nov 1, 2021
@jaredpalmer jaredpalmer reopened this Nov 15, 2021
@jaredpalmer jaredpalmer deleted the refactor-to-pipeline branch December 1, 2021 15:04
sokra added a commit that referenced this pull request Oct 25, 2022
explain variables using multiple lines when needed
sokra added a commit that referenced this pull request Oct 25, 2022
* update swc for nested css bugfixes
* add test cases

Not working yet: `@import` of CSS Modules.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant