import { DEFAULT_BARS } from '../../../components/pages/pack-home/PackBenchmarks'
import { DocsBenchmarksGraph } from '../../../components/pages/pack-home/DocsBenchmarksGraph';
import Callout from '../../../components/Callout';
import { ThemedImageFigure } from '../../../components/image/ThemedImageFigure';
import { HMR_BARS } from '../../../components/pages/pack-home/PackBenchmarks'

# Benchmarks

We created a test generator that makes an application with a variable amount of modules to benchmark cold startup and file updating tasks. This generated app includes entries for these tools:

- Next.js 11
- Next.js 12
- Next.js 13 with Turbopack
- Vite

As the current state of the art, we're including [Vite](https://vitejs.dev/) along with webpack-based [Next.js](https://nextjs.org) solutions. All of these toolchains point to the same generated component tree, assembling a [Sierpiński triangle](https://en.wikipedia.org/wiki/Sierpi%C5%84ski_triangle) in the browser, where every triangle is a separate module.

<ThemedImageFigure
  borderRadius={false}
  dark={{
    source: '/images/blog/turbopack-benchmarks/triangle-dark.png',
    height: 600,
    width: 1200
  }}
  light={{
    source: '/images/blog/turbopack-benchmarks/triangle-light.png',
    height: 600,
    width: 1200
  }}
  captionSpacing={-12}
  caption="This image is a screenshot of the test application we run our benchmarks on. It depicts a Sierpiński triangle where each single triangle is its own component, separated in its own file. In this example, there are 3,000 triangles being rendered to the screen."
/>

## Metrics

Let's break down exactly what each of these metrics mean, and how they'll impact your day-to-day developer experience.

<Callout type="info">
  Curious about how these benchmarks are implemented, or want to run them yourself? Check out [our benchmark suite documentation in the Turbo monorepo](https://github.com/vercel/turbo/blob/main/crates/next-dev/benches/README.md).
</Callout>

### Cold startup time

This test measures how fast a local development server starts up on an application of various sizes. We measure this as the time from startup (without cache) until the app is rendered in the browser. We do not wait for the app to be interactive or hydrated in the browser for this dataset.

<DocsBenchmarksGraph category="cold" bars={DEFAULT_BARS} />

<ThemedImageFigure
  borderRadius={true}
  dark={{
    source: '/images/blog/turbopack-benchmarks/bench_startup_dark.svg',
    height: 720,
    width: 1960
  }}
  light={{
    source: '/images/blog/turbopack-benchmarks/bench_startup_light.svg',
    height: 720,
    width: 1960
  }}
  captionSpacing={24}
  caption="Startup time by module count. Benchmark data generated from 16” MacBook Pro 2021, M1 Max, 32GB RAM, macOS 13.0.1 (22A400)."
/>

#### Data

To run this benchmark yourself, clone [`vercel/turbo`](https://github.com/vercel/turbo) and then use this command from the root:

```sh
TURBOPACK_BENCH_COUNTS=1000,5000,10000,30000 TURBOPACK_BENCH_BUNDLERS=all cargo bench -p next-dev "startup/(Turbopack SSR|Next.js 12 SSR|Next.js 11 SSR|Vite SWC CSR)."
```

Here are the numbers we were able to produce on a 16” MacBook Pro 2021, M1 Max, 32GB RAM, macOS 13.0.1 (22A400):

```sh
bench_startup/Next.js 11 SSR/1000 modules                  9.2±0.04s
bench_startup/Next.js 11 SSR/5000 modules                 32.9±0.67s
bench_startup/Next.js 11 SSR/10000 modules                71.8±2.57s
bench_startup/Next.js 11 SSR/30000 modules               237.6±6.43s
bench_startup/Next.js 12 SSR/1000 modules                  3.6±0.02s
bench_startup/Next.js 12 SSR/5000 modules                 12.1±0.32s
bench_startup/Next.js 12 SSR/10000 modules                23.3±0.32s
bench_startup/Next.js 12 SSR/30000 modules                89.1±0.21s
bench_startup/Turbopack SSR/1000 modules               1381.9±5.62ms
bench_startup/Turbopack SSR/5000 modules                   4.0±0.04s
bench_startup/Turbopack SSR/10000 modules                  7.3±0.07s
bench_startup/Turbopack SSR/30000 modules                 22.0±0.32s
bench_startup/Vite SWC CSR/1000 modules                    4.2±0.02s
bench_startup/Vite SWC CSR/5000 modules                   16.6±0.08s
bench_startup/Vite SWC CSR/10000 modules                  32.3±0.12s
bench_startup/Vite SWC CSR/30000 modules                  97.7±1.53s
```

### File updates (HMR)

We also measure how quickly the development server works from when an update is applied to a source file to when the corresponding change is re-rendered in the browser.

For Hot Module Reloading (HMR) benchmarks, we first start the dev server on a fresh installation with the test application. We wait for the HMR server to boot up by running updates until one succeeds. We then run ten changes to warm up the tooling. This step is important as it prevents discrepancies that can arise with cold processes.

Once our tooling is warmed up, we run a series of updates to a list of modules within the test application. Modules are sampled randomly with a distribution that ensures we test a uniform number of modules per module depth. The depth of a module is its distance from the entry module in the dependency graph. For instance, if the entry module A imports module B, which imports modules C and D, the depth of the entry module A will be 0, that of module B will be 1, and that of modules C and D will be 2. Modules A and B will have an equal probability of being sampled, but modules C and D will only have half the probability of being sampled.

We report the linear regression slope of the data points as the target metric. This is an estimate of the average time it takes for the tooling to apply an update to the application.

<DocsBenchmarksGraph category="file_change" bars={HMR_BARS} />

<ThemedImageFigure
  borderRadius={true}
  dark={{
    source: '/images/blog/turbopack-benchmarks/bench_hmr_to_commit_dark.svg',
    height: 720,
    width: 1960
  }}
  light={{
    source: '/images/blog/turbopack-benchmarks/bench_hmr_to_commit_light.svg',
    height: 720,
    width: 1960
  }}
  captionSpacing={24}
  caption="Turbopack, Next.js (webpack), and Vite HMR by module count. Benchmark data generated from 16” MacBook Pro 2021, M1 Max, 32GB RAM, macOS 13.0.1 (22A400)."
/>

<a id="bench"/>

<ThemedImageFigure
  borderRadius={true}
  dark={{
    source: '/images/blog/turbopack-benchmarks/bench_hmr_to_commit_turbopack_vite_dark.svg',
    height: 720,
    width: 1960
  }}
  light={{
    source: '/images/blog/turbopack-benchmarks/bench_hmr_to_commit_turbopack_vite_light.svg',
    height: 720,
    width: 1960
  }}
  captionSpacing={24}
  caption="Turbopack and Vite HMR by module count. Benchmark data generated from 16” MacBook Pro 2021, M1 Max, 32GB RAM, macOS 13.0.1 (22A400)."
/>

The takeaway from these benchmarks: Turbopack performance is a function of **the size of an update**, not the size of an application.

We're excited by Turbopack's performance, but we also expect to make further speed improvements for both small and large applications in the future. For more info, visit the comparison docs for [Vite](/pack/docs/comparisons/vite) and [webpack](/pack/docs/comparisons/webpack).

#### Data

To run this benchmark yourself, clone [`vercel/turbo`](https://github.com/vercel/turbo) and then use this command from the root:

```
TURBOPACK_BENCH_COUNTS=1000,5000,10000,30000 TURBOPACK_BENCH_BUNDLERS=all cargo bench -p next-dev "hmr_to_commit/(Turbopack SSR|Next.js 12 SSR|Next.js 11 SSR|Vite SWC CSR)"
```

Here are the numbers we were able to produce on a 16” MacBook Pro 2021, M1 Max, 32GB RAM, macOS 13.0.1 (22A400):

```sh
bench_hmr_to_commit/Next.js 11 SSR/1000 modules         211.6±1.14ms
bench_hmr_to_commit/Next.js 11 SSR/5000 modules        866.0±34.44ms
bench_hmr_to_commit/Next.js 11 SSR/10000 modules           2.4±0.13s
bench_hmr_to_commit/Next.js 11 SSR/30000 modules           9.5±3.12s
bench_hmr_to_commit/Next.js 12 SSR/1000 modules         146.2±2.17ms
bench_hmr_to_commit/Next.js 12 SSR/5000 modules        494.7±25.13ms
bench_hmr_to_commit/Next.js 12 SSR/10000 modules     1151.9±280.68ms
bench_hmr_to_commit/Next.js 12 SSR/30000 modules           6.4±2.29s
bench_hmr_to_commit/Turbopack SSR/1000 modules           18.9±2.92ms
bench_hmr_to_commit/Turbopack SSR/5000 modules           23.8±0.31ms
bench_hmr_to_commit/Turbopack SSR/10000 modules          23.0±0.35ms
bench_hmr_to_commit/Turbopack SSR/30000 modules          22.5±0.88ms
bench_hmr_to_commit/Vite SWC CSR/1000 modules           104.8±1.52ms
bench_hmr_to_commit/Vite SWC CSR/5000 modules           109.6±3.94ms
bench_hmr_to_commit/Vite SWC CSR/10000 modules          113.0±1.20ms
bench_hmr_to_commit/Vite SWC CSR/30000 modules         133.3±23.65ms
```

Note that Vite is using the official [SWC plugin](https://github.com/vitejs/vite-plugin-react-swc) for these benchmarks, which is not the default configuration.

If you have questions about the benchmark, please open an [issue on GitHub](https://github.com/vercel/turbo/issues).
