这是indexloc提供的服务,不要输入任何密码
Skip to content

ci(workflow): format integration test results #3534

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 1, 2023
Merged

Conversation

kwonoj
Copy link
Contributor

@kwonoj kwonoj commented Jan 30, 2023

Closes WEB-486.

This PR amends stat report action formatting - mainly introducing diff summary between base (main for the PR, release to release for new next.js release). This summary will be piped into slack channel once this PR lands.

There maybe some things to amend further, but primary logic seems to working. Please note current base (main) results are incorrectly collected, makes diff summary looks weird. We may need wipe out incorrect data.

@vercel
Copy link

vercel bot commented Jan 30, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated
examples-designsystem-docs 🔄 Building (Inspect) Jan 31, 2023 at 9:27PM (UTC)
turbo-site ✅ Ready (Inspect) Visit Preview 💬 Add your feedback Jan 31, 2023 at 9:27PM (UTC)
8 Ignored Deployments
Name Status Preview Comments Updated
examples-basic-web ⬜️ Ignored (Inspect) Jan 31, 2023 at 9:27PM (UTC)
examples-cra-web ⬜️ Ignored (Inspect) Jan 31, 2023 at 9:27PM (UTC)
examples-kitchensink-blog ⬜️ Ignored (Inspect) Jan 31, 2023 at 9:27PM (UTC)
examples-native-web ⬜️ Ignored (Inspect) Jan 31, 2023 at 9:27PM (UTC)
examples-nonmonorepo ⬜️ Ignored (Inspect) Jan 31, 2023 at 9:27PM (UTC)
examples-svelte-web ⬜️ Ignored (Inspect) Jan 31, 2023 at 9:27PM (UTC)
examples-tailwind-web ⬜️ Ignored (Inspect) Jan 31, 2023 at 9:27PM (UTC)
examples-vite-web ⬜️ Ignored (Inspect) Jan 31, 2023 at 9:27PM (UTC)

@kwonoj kwonoj force-pushed the test-results-diff branch 2 times, most recently from 79b959d to 6c64b04 Compare January 30, 2023 21:20
@vercel vercel deleted a comment from github-actions bot Jan 30, 2023
@vercel vercel deleted a comment from github-actions bot Jan 30, 2023
@kwonoj kwonoj force-pushed the test-results-diff branch 3 times, most recently from 8419b45 to 7064c2d Compare January 30, 2023 23:45
@vercel vercel deleted a comment from github-actions bot Jan 31, 2023
@kwonoj kwonoj force-pushed the test-results-diff branch from 7064c2d to 3e7f116 Compare January 31, 2023 00:24
@github-actions
Copy link
Contributor

Benchmark for b688e73

Click to view benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 9137.85µs ± 92.72µs 9136.00µs ± 59.96µs -0.02%
bench_hmr_to_commit/Turbopack RCC/1000 modules 9535.12µs ± 117.63µs 9643.40µs ± 80.65µs +1.14%
bench_hmr_to_commit/Turbopack RSC/1000 modules 500.11ms ± 2.67ms 492.42ms ± 2.93ms -1.54%
bench_hmr_to_commit/Turbopack SSR/1000 modules 9393.11µs ± 65.32µs 9349.10µs ± 103.91µs -0.47%
bench_hmr_to_eval/Turbopack CSR/1000 modules 8255.80µs ± 41.10µs 8266.74µs ± 55.90µs +0.13%
bench_hmr_to_eval/Turbopack RCC/1000 modules 8468.30µs ± 66.27µs 8404.46µs ± 53.72µs -0.75%
bench_hmr_to_eval/Turbopack SSR/1000 modules 8443.35µs ± 69.98µs 8488.57µs ± 70.70µs +0.54%
bench_hydration/Turbopack RCC/1000 modules 3792.25ms ± 13.89ms 3779.76ms ± 11.29ms -0.33%
bench_hydration/Turbopack RSC/1000 modules 3298.17ms ± 9.16ms 3316.19ms ± 8.39ms +0.55%
bench_hydration/Turbopack SSR/1000 modules 3102.15ms ± 10.90ms 3120.50ms ± 12.95ms +0.59%
bench_startup/Turbopack CSR/1000 modules 2107.38ms ± 12.68ms 2139.91ms ± 15.84ms +1.54%
bench_startup/Turbopack RCC/1000 modules 2529.70ms ± 10.50ms 2518.77ms ± 3.95ms -0.43%
bench_startup/Turbopack RSC/1000 modules 2423.28ms ± 8.13ms 2440.30ms ± 5.70ms +0.70%
bench_startup/Turbopack SSR/1000 modules 2060.46ms ± 4.58ms 2076.36ms ± 4.67ms +0.77%

@kwonoj kwonoj force-pushed the test-results-diff branch 7 times, most recently from fae6420 to c2ee236 Compare January 31, 2023 03:04
@github-actions
Copy link
Contributor

Benchmark for 7fdc2f1

Click to view benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 9222.10µs ± 73.82µs 9154.49µs ± 69.67µs -0.73%
bench_hmr_to_commit/Turbopack RCC/1000 modules 9617.88µs ± 78.24µs 9610.29µs ± 88.79µs -0.08%
bench_hmr_to_commit/Turbopack RSC/1000 modules 481.54ms ± 1.19ms 476.53ms ± 2.54ms -1.04%
bench_hmr_to_commit/Turbopack SSR/1000 modules 9359.57µs ± 89.58µs 9422.66µs ± 82.99µs +0.67%
bench_hmr_to_eval/Turbopack CSR/1000 modules 8199.13µs ± 51.41µs 8188.66µs ± 63.48µs -0.13%
bench_hmr_to_eval/Turbopack RCC/1000 modules 8461.16µs ± 55.55µs 8349.44µs ± 81.70µs -1.32%
bench_hmr_to_eval/Turbopack SSR/1000 modules 8485.20µs ± 91.11µs 8312.94µs ± 68.82µs -2.03%
bench_hydration/Turbopack RCC/1000 modules 3781.61ms ± 7.79ms 3754.19ms ± 12.24ms -0.73%
bench_hydration/Turbopack RSC/1000 modules 3301.97ms ± 13.98ms 3300.00ms ± 7.46ms -0.06%
bench_hydration/Turbopack SSR/1000 modules 3069.73ms ± 10.86ms 3057.44ms ± 9.94ms -0.40%
bench_startup/Turbopack CSR/1000 modules 2092.09ms ± 10.92ms 2111.73ms ± 13.42ms +0.94%
bench_startup/Turbopack RCC/1000 modules 2503.85ms ± 8.34ms 2515.37ms ± 8.14ms +0.46%
bench_startup/Turbopack RSC/1000 modules 2440.82ms ± 8.05ms 2459.20ms ± 10.00ms +0.75%
bench_startup/Turbopack SSR/1000 modules 2075.05ms ± 6.03ms 2064.28ms ± 8.82ms -0.52%

@kwonoj kwonoj force-pushed the test-results-diff branch 5 times, most recently from 5e59955 to 61092de Compare January 31, 2023 06:50
@vercel vercel deleted a comment from github-actions bot Jan 31, 2023
@github-actions
Copy link
Contributor

Benchmark for e3b53af

Click to view benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 9027.70µs ± 61.05µs 9133.51µs ± 58.44µs +1.17%
bench_hmr_to_commit/Turbopack RCC/1000 modules 9360.87µs ± 105.65µs 9353.02µs ± 57.24µs -0.08%
bench_hmr_to_commit/Turbopack RSC/1000 modules 476.40ms ± 2.99ms 479.49ms ± 2.40ms +0.65%
bench_hmr_to_commit/Turbopack SSR/1000 modules 9270.20µs ± 57.33µs 9310.05µs ± 61.30µs +0.43%
bench_hmr_to_eval/Turbopack CSR/1000 modules 8109.26µs ± 73.67µs 8104.29µs ± 82.03µs -0.06%
bench_hmr_to_eval/Turbopack RCC/1000 modules 8466.14µs ± 66.76µs 8464.82µs ± 72.55µs -0.02%
bench_hmr_to_eval/Turbopack SSR/1000 modules 8377.08µs ± 75.15µs 8405.48µs ± 96.26µs +0.34%
bench_hydration/Turbopack RCC/1000 modules 3730.34ms ± 12.23ms 3740.14ms ± 11.38ms +0.26%
bench_hydration/Turbopack RSC/1000 modules 3296.29ms ± 12.25ms 3284.59ms ± 12.18ms -0.35%
bench_hydration/Turbopack SSR/1000 modules 3040.65ms ± 12.57ms 3024.93ms ± 11.42ms -0.52%
bench_startup/Turbopack CSR/1000 modules 2050.79ms ± 11.61ms 2049.71ms ± 12.06ms -0.05%
bench_startup/Turbopack RCC/1000 modules 2519.43ms ± 8.71ms 2503.03ms ± 8.13ms -0.65%
bench_startup/Turbopack RSC/1000 modules 2371.21ms ± 7.17ms 2391.66ms ± 7.48ms +0.86%
bench_startup/Turbopack SSR/1000 modules 2012.60ms ± 5.09ms 2004.72ms ± 4.21ms -0.39%

@kwonoj kwonoj force-pushed the test-results-diff branch 2 times, most recently from d947546 to 9b1c43c Compare January 31, 2023 08:21
@kwonoj kwonoj force-pushed the test-results-diff branch from 9b1c43c to 53babbe Compare January 31, 2023 08:52
@github-actions
Copy link
Contributor

Benchmark for bd97594

Click to view benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 9614.41µs ± 61.91µs 9806.15µs ± 85.09µs +1.99%
bench_hmr_to_commit/Turbopack RCC/1000 modules 10.07ms ± 0.11ms 10.07ms ± 0.08ms +0.02%
bench_hmr_to_commit/Turbopack RSC/1000 modules 482.45ms ± 3.01ms 480.45ms ± 2.61ms -0.41%
bench_hmr_to_commit/Turbopack SSR/1000 modules 9936.99µs ± 107.21µs 9924.53µs ± 95.18µs -0.13%
bench_hmr_to_eval/Turbopack CSR/1000 modules 8811.95µs ± 68.69µs 8725.21µs ± 66.83µs -0.98%
bench_hmr_to_eval/Turbopack RCC/1000 modules 8946.54µs ± 75.78µs 8952.90µs ± 77.08µs +0.07%
bench_hmr_to_eval/Turbopack SSR/1000 modules 8935.23µs ± 78.80µs 8929.06µs ± 66.88µs -0.07%
bench_hydration/Turbopack RCC/1000 modules 3797.35ms ± 13.15ms 3821.67ms ± 14.90ms +0.64%
bench_hydration/Turbopack RSC/1000 modules 3313.79ms ± 14.58ms 3342.78ms ± 7.91ms +0.88%
bench_hydration/Turbopack SSR/1000 modules 3091.19ms ± 14.02ms 3101.35ms ± 11.44ms +0.33%
bench_startup/Turbopack CSR/1000 modules 2076.43ms ± 6.22ms 2079.01ms ± 12.02ms +0.12%
bench_startup/Turbopack RCC/1000 modules 2507.14ms ± 7.03ms 2515.69ms ± 8.37ms +0.34%
bench_startup/Turbopack RSC/1000 modules 2400.96ms ± 7.47ms 2419.13ms ± 5.18ms +0.76%
bench_startup/Turbopack SSR/1000 modules 2043.79ms ± 4.86ms 2044.14ms ± 5.92ms +0.02%

@github-actions
Copy link
Contributor

github-actions bot commented Jan 31, 2023

⚠️ CI failed ⚠️

The following steps have failed in CI:

  • Rust tests
  • Rust benchmark tests

See workflow summary for details

@kwonoj kwonoj force-pushed the test-results-diff branch from 53babbe to 4ae88f2 Compare January 31, 2023 15:45
@github-actions
Copy link
Contributor

github-actions bot commented Jan 31, 2023

Failing next.js integration test suites

Commit: d6b21e2

Test summary

main (752319a / v13.1.7-canary.1) Current (d6b21e2 / v13.1.7-canary.1) Diff
Failed Suites 3 12 🔼 9
Failed Cases 9 38 🔼 29

Fixed tests:
- test/integration/catches-missing-getStaticProps/test/index.test.js
Newly failed tests:
- test/development/acceptance-app/component-stack.test.ts
- test/e2e/404-page-router/index.test.ts
- test/integration/404-page-app/test/index.test.js
- test/integration/bigint/test/index.test.js
- test/development/acceptance-app/component-stack.test.ts
- test/e2e/i18n-ignore-rewrite-source-locale/rewrites-with-basepath.test.ts
- test/integration/404-page-app/test/index.test.js
- test/e2e/app-dir/app-alias/app-alias.test.ts
- test/integration/bigint/test/index.test.js

Full test report

test/development/acceptance-app/component-stack.test.ts

  • Legacy decorators SWC option > should compile with legacy decorators enabled
Expand output

● Legacy decorators SWC option › should compile with legacy decorators enabled

page.waitForSelector: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator('#count')
============================================================

  329 |     return this.chain(() => {
  330 |       return page
> 331 |         .waitForSelector(selector, { timeout, state: 'attached' })
      |          ^
  332 |         .then(async (el) => {
  333 |           // it seems selenium waits longer and tests rely on this behavior
  334 |           // so we wait for the load event fire before returning

  at lib/browsers/playwright.ts:331:10

test/e2e/404-page-router/index.test.ts

  • manual-client-base-path > should not update with basePath on mount /
  • manual-client-base-path > should not update with basePath on mount /another
  • manual-client-base-path > should not update with basePath on mount /dynamic/first
  • manual-client-base-path > should not update with basePath on mount /dynamic/second
  • manual-client-base-path > should navigate correctly from index
  • manual-client-base-path > should navigate correctly from another
Expand output

● manual-client-base-path › should not update with basePath on mount /

TIMED OUT: success

undefined

  559 |
  560 |   if (hardError) {
> 561 |     throw new Error('TIMED OUT: ' + regex + '\n\n' + content)
      |           ^
  562 |   }
  563 |   return false
  564 | }

  at check (lib/next-test-utils.js:561:11)
  at Object.<anonymous> (e2e/manual-client-base-path/index.test.ts:121:7)

● manual-client-base-path › should not update with basePath on mount /another

TIMED OUT: success

undefined

  559 |
  560 |   if (hardError) {
> 561 |     throw new Error('TIMED OUT: ' + regex + '\n\n' + content)
      |           ^
  562 |   }
  563 |   return false
  564 | }

  at check (lib/next-test-utils.js:561:11)
  at Object.<anonymous> (e2e/manual-client-base-path/index.test.ts:121:7)

● manual-client-base-path › should not update with basePath on mount /dynamic/first

TIMED OUT: success

undefined

  559 |
  560 |   if (hardError) {
> 561 |     throw new Error('TIMED OUT: ' + regex + '\n\n' + content)
      |           ^
  562 |   }
  563 |   return false
  564 | }

  at check (lib/next-test-utils.js:561:11)
  at Object.<anonymous> (e2e/manual-client-base-path/index.test.ts:121:7)

● manual-client-base-path › should not update with basePath on mount /dynamic/second

TIMED OUT: success

undefined

  559 |
  560 |   if (hardError) {
> 561 |     throw new Error('TIMED OUT: ' + regex + '\n\n' + content)
      |           ^
  562 |   }
  563 |   return false
  564 | }

  at check (lib/next-test-utils.js:561:11)
  at Object.<anonymous> (e2e/manual-client-base-path/index.test.ts:121:7)

● manual-client-base-path › should navigate correctly from index

page.waitForSelector: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator('#to-another')
============================================================

  329 |     return this.chain(() => {
  330 |       return page
> 331 |         .waitForSelector(selector, { timeout, state: 'attached' })
      |          ^
  332 |         .then(async (el) => {
  333 |           // it seems selenium waits longer and tests rely on this behavior
  334 |           // so we wait for the load event fire before returning

  at lib/browsers/playwright.ts:331:10
  at Object.<anonymous> (e2e/manual-client-base-path/index.test.ts:146:5)

● manual-client-base-path › should navigate correctly from another

page.waitForSelector: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator('#to-index')
============================================================

  329 |     return this.chain(() => {
  330 |       return page
> 331 |         .waitForSelector(selector, { timeout, state: 'attached' })
      |          ^
  332 |         .then(async (el) => {
  333 |           // it seems selenium waits longer and tests rely on this behavior
  334 |           // so we wait for the load event fire before returning

  at lib/browsers/playwright.ts:331:10
  at Object.<anonymous> (e2e/manual-client-base-path/index.test.ts:185:5)

test/integration/404-page-app/test/index.test.js

  • 404 Page Support with _app > dev mode > should not show pages/404 GIP error if _app has GIP
Expand output

● 404 Page Support with _app › dev mode › should not show pages/404 GIP error if _app has GIP

expect(received).toBe(expected) // Object.is equality

Expected: 404
Received: 500

  79 |     it('should not show pages/404 GIP error if _app has GIP', async () => {
  80 |       const res = await fetchViaHTTP(appPort, '/abc')
> 81 |       expect(res.status).toBe(404)
     |                          ^
  82 |       const $ = cheerio.load(await res.text())
  83 |       expect($('#404-title').text()).toBe('Hi There')
  84 |       expect(stderr).not.toMatch(gip404Err)

  at Object.<anonymous> (integration/404-page-app/test/index.test.js:81:26)

test/development/acceptance/ReactRefresh.test.ts

  • ReactRefresh > can edit a component without losing state
  • ReactRefresh > cyclic dependencies
Expand output

● ReactRefresh › can edit a component without losing state

Application is in inconsistent state: timeout.

  82 |           }
  83 |           if (status !== 'pending') {
> 84 |             throw new Error(`Application is in inconsistent state: ${status}.`)
     |                   ^
  85 |           }
  86 |
  87 |           await new Promise((resolve) => setTimeout(resolve, 30))

  at Object.patch (development/acceptance/helpers.ts:84:19)
  at Object.<anonymous> (development/acceptance/ReactRefresh.test.ts:20:5)

● ReactRefresh › cyclic dependencies

Application is in inconsistent state: timeout.

  82 |           }
  83 |           if (status !== 'pending') {
> 84 |             throw new Error(`Application is in inconsistent state: ${status}.`)
     |                   ^
  85 |           }
  86 |
  87 |           await new Promise((resolve) => setTimeout(resolve, 30))

  at Object.patch (development/acceptance/helpers.ts:84:19)
  at Object.<anonymous> (development/acceptance/ReactRefresh.test.ts:157:5)

test/e2e/app-dir/app-rendering/rendering.test.ts

  • app dir rendering > should serve app/page.server.js at /
  • app dir rendering > ISR > should revalidate the page when revalidate is configured
  • app dir rendering > SSR only > should run data in layout and page
  • app dir rendering > SSR only > should run data fetch in parallel
  • app dir rendering > static only > should run data in layout and page
  • app dir rendering > static only > should run data in parallel during development
Expand output

● app dir rendering › should serve app/page.server.js at /

expect(received).toContain(expected) // indexOf

Expected substring: "app/page.server.js"
Received string:    "failed to canonicalize embedded file path·
Caused by:
    No such file or directory (os error 2)"

  12 |     it('should serve app/page.server.js at /', async () => {
  13 |       const html = await next.render('/')
> 14 |       expect(html).toContain('app/page.server.js')
     |                    ^
  15 |     })
  16 |
  17 |     describe('SSR only', () => {

  at Object.<anonymous> (e2e/app-dir/app-rendering/rendering.test.ts:14:20)

● app dir rendering › SSR only › should run data in layout and page

expect(received).toBe(expected) // Object.is equality

Expected: "hello from layout"
Received: ""

  18 |       it('should run data in layout and page', async () => {
  19 |         const $ = await next.render$('/ssr-only/nested')
> 20 |         expect($('#layout-message').text()).toBe('hello from layout')
     |                                             ^
  21 |         expect($('#page-message').text()).toBe('hello from page')
  22 |       })
  23 |

  at Object.<anonymous> (e2e/app-dir/app-rendering/rendering.test.ts:20:45)

● app dir rendering › SSR only › should run data fetch in parallel

expect(received).toBe(expected) // Object.is equality

Expected: "hello from slow layout"
Received: ""

  30 |         // Using 7 seconds to ensure external factors causing slight slowness don't fail the tests
  31 |         expect(duration < 7000).toBe(true)
> 32 |         expect($('#slow-layout-message').text()).toBe('hello from slow layout')
     |                                                  ^
  33 |         expect($('#slow-page-message').text()).toBe('hello from slow page')
  34 |       })
  35 |     })

  at Object.<anonymous> (e2e/app-dir/app-rendering/rendering.test.ts:32:50)

● app dir rendering › static only › should run data in layout and page

expect(received).toBe(expected) // Object.is equality

Expected: "hello from layout"
Received: ""

  38 |       it('should run data in layout and page', async () => {
  39 |         const $ = await next.render$('/static-only/nested')
> 40 |         expect($('#layout-message').text()).toBe('hello from layout')
     |                                             ^
  41 |         expect($('#page-message').text()).toBe('hello from page')
  42 |       })
  43 |

  at Object.<anonymous> (e2e/app-dir/app-rendering/rendering.test.ts:40:45)

● app dir rendering › static only › should run data in parallel during development

expect(received).toBe(expected) // Object.is equality

Expected: "hello from slow layout"
Received: ""

  54 |         // expect(duration < (isDev ? 7000 : 2000)).toBe(true)
  55 |         // expect(duration < 7000).toBe(true)
> 56 |         expect($('#slow-layout-message').text()).toBe('hello from slow layout')
     |                                                  ^
  57 |         expect($('#slow-page-message').text()).toBe('hello from slow page')
  58 |       })
  59 |     })

  at Object.<anonymous> (e2e/app-dir/app-rendering/rendering.test.ts:56:50)

● app dir rendering › ISR › should revalidate the page when revalidate is configured

expect(received).toBe(expected) // Object.is equality

Expected: "hello from layout"
Received: ""

  71 |         }
  72 |         const { $ } = await getPage()
> 73 |         expect($('#layout-message').text()).toBe('hello from layout')
     |                                             ^
  74 |         expect($('#page-message').text()).toBe('hello from page')
  75 |
  76 |         const layoutNow = $('#layout-now').text()

  at Object.<anonymous> (e2e/app-dir/app-rendering/rendering.test.ts:73:45)

test/integration/bigint/test/index.test.js

  • bigint API route support > dev mode > should return 200
  • bigint API route support > dev mode > should return the BigInt result text
Expand output

● bigint API route support › dev mode › should return 200

expect(received).toEqual(expected) // deep equality

Expected: 200
Received: 500

  22 |       method: 'GET',
  23 |     })
> 24 |     expect(res.status).toEqual(200)
     |                        ^
  25 |   })
  26 |
  27 |   it('should return the BigInt result text', async () => {

  at Object.<anonymous> (integration/bigint/test/index.test.js:24:24)

● bigint API route support › dev mode › should return the BigInt result text

expect(received).toEqual(expected) // deep equality

Expected: "3"
Received: false

  29 |       method: 'GET',
  30 |     }).then((res) => res.ok && res.text())
> 31 |     expect(resText).toEqual('3')
     |                     ^
  32 |   })
  33 | }
  34 |

  at Object.<anonymous> (integration/bigint/test/index.test.js:31:21)

test/development/acceptance-app/component-stack.test.ts

  • Legacy decorators SWC option > should compile with legacy decorators enabled
Expand output

● Legacy decorators SWC option › should compile with legacy decorators enabled

page.waitForSelector: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator('#count')
============================================================

  329 |     return this.chain(() => {
  330 |       return page
> 331 |         .waitForSelector(selector, { timeout, state: 'attached' })
      |          ^
  332 |         .then(async (el) => {
  333 |           // it seems selenium waits longer and tests rely on this behavior
  334 |           // so we wait for the load event fire before returning

  at lib/browsers/playwright.ts:331:10

test/e2e/i18n-ignore-rewrite-source-locale/rewrites-with-basepath.test.ts

  • i18n-ignore-rewrite-source-locale with basepath > get public file by skipping locale in rewrite, locale:
  • i18n-ignore-rewrite-source-locale with basepath > get public file by skipping locale in rewrite, locale: /en
  • i18n-ignore-rewrite-source-locale with basepath > get public file by skipping locale in rewrite, locale: /sv
  • i18n-ignore-rewrite-source-locale with basepath > get public file by skipping locale in rewrite, locale: /nl
  • i18n-ignore-rewrite-source-locale with basepath > call api by skipping locale in rewrite, locale:
  • i18n-ignore-rewrite-source-locale with basepath > call api by skipping locale in rewrite, locale: /en
  • i18n-ignore-rewrite-source-locale with basepath > call api by skipping locale in rewrite, locale: /sv
  • i18n-ignore-rewrite-source-locale with basepath > call api by skipping locale in rewrite, locale: /nl
  • i18n-ignore-rewrite-source-locale with basepath > get _next/static/ files by skipping locale in rewrite, locale:
  • i18n-ignore-rewrite-source-locale with basepath > get _next/static/ files by skipping locale in rewrite, locale: /en
  • i18n-ignore-rewrite-source-locale with basepath > get _next/static/ files by skipping locale in rewrite, locale: /sv
  • i18n-ignore-rewrite-source-locale with basepath > get _next/static/ files by skipping locale in rewrite, locale: /nl
Expand output

● i18n-ignore-rewrite-source-locale with basepath › get public file by skipping locale in rewrite, locale:

expect(received).toContain(expected) // indexOf

Expected substring: "hello from file.txt"
Received string:    "failed to canonicalize embedded file path·
Caused by:
    No such file or directory (os error 2)"

  56 |         `/basepath${locale}/rewrite-files/file.txt`
  57 |       )
> 58 |       expect(res).toContain('hello from file.txt')
     |                   ^
  59 |     }
  60 |   )
  61 |

  at e2e/i18n-ignore-rewrite-source-locale/rewrites-with-basepath.test.ts:58:19

● i18n-ignore-rewrite-source-locale with basepath › get public file by skipping locale in rewrite, locale: /en

expect(received).toContain(expected) // indexOf

Expected substring: "hello from file.txt"
Received string:    "failed to canonicalize embedded file path·
Caused by:
    No such file or directory (os error 2)"

  56 |         `/basepath${locale}/rewrite-files/file.txt`
  57 |       )
> 58 |       expect(res).toContain('hello from file.txt')
     |                   ^
  59 |     }
  60 |   )
  61 |

  at e2e/i18n-ignore-rewrite-source-locale/rewrites-with-basepath.test.ts:58:19

● i18n-ignore-rewrite-source-locale with basepath › get public file by skipping locale in rewrite, locale: /sv

expect(received).toContain(expected) // indexOf

Expected substring: "hello from file.txt"
Received string:    "failed to canonicalize embedded file path·
Caused by:
    No such file or directory (os error 2)"

  56 |         `/basepath${locale}/rewrite-files/file.txt`
  57 |       )
> 58 |       expect(res).toContain('hello from file.txt')
     |                   ^
  59 |     }
  60 |   )
  61 |

  at e2e/i18n-ignore-rewrite-source-locale/rewrites-with-basepath.test.ts:58:19

● i18n-ignore-rewrite-source-locale with basepath › get public file by skipping locale in rewrite, locale: /nl

expect(received).toContain(expected) // indexOf

Expected substring: "hello from file.txt"
Received string:    "failed to canonicalize embedded file path·
Caused by:
    No such file or directory (os error 2)"

  56 |         `/basepath${locale}/rewrite-files/file.txt`
  57 |       )
> 58 |       expect(res).toContain('hello from file.txt')
     |                   ^
  59 |     }
  60 |   )
  61 |

  at e2e/i18n-ignore-rewrite-source-locale/rewrites-with-basepath.test.ts:58:19

● i18n-ignore-rewrite-source-locale with basepath › call api by skipping locale in rewrite, locale:

expect(received).toContain(expected) // indexOf

Expected substring: "hello from api"
Received string:    "failed to canonicalize embedded file path·
Caused by:
    No such file or directory (os error 2)"

  67 |         `/basepath${locale}/rewrite-api/hello`
  68 |       )
> 69 |       expect(res).toContain('hello from api')
     |                   ^
  70 |     }
  71 |   )
  72 |

  at e2e/i18n-ignore-rewrite-source-locale/rewrites-with-basepath.test.ts:69:19

● i18n-ignore-rewrite-source-locale with basepath › call api by skipping locale in rewrite, locale: /en

expect(received).toContain(expected) // indexOf

Expected substring: "hello from api"
Received string:    "failed to canonicalize embedded file path·
Caused by:
    No such file or directory (os error 2)"

  67 |         `/basepath${locale}/rewrite-api/hello`
  68 |       )
> 69 |       expect(res).toContain('hello from api')
     |                   ^
  70 |     }
  71 |   )
  72 |

  at e2e/i18n-ignore-rewrite-source-locale/rewrites-with-basepath.test.ts:69:19

● i18n-ignore-rewrite-source-locale with basepath › call api by skipping locale in rewrite, locale: /sv

expect(received).toContain(expected) // indexOf

Expected substring: "hello from api"
Received string:    "failed to canonicalize embedded file path·
Caused by:
    No such file or directory (os error 2)"

  67 |         `/basepath${locale}/rewrite-api/hello`
  68 |       )
> 69 |       expect(res).toContain('hello from api')
     |                   ^
  70 |     }
  71 |   )
  72 |

  at e2e/i18n-ignore-rewrite-source-locale/rewrites-with-basepath.test.ts:69:19

● i18n-ignore-rewrite-source-locale with basepath › call api by skipping locale in rewrite, locale: /nl

expect(received).toContain(expected) // indexOf

Expected substring: "hello from api"
Received string:    "failed to canonicalize embedded file path·
Caused by:
    No such file or directory (os error 2)"

  67 |         `/basepath${locale}/rewrite-api/hello`
  68 |       )
> 69 |       expect(res).toContain('hello from api')
     |                   ^
  70 |     }
  71 |   )
  72 |

  at e2e/i18n-ignore-rewrite-source-locale/rewrites-with-basepath.test.ts:69:19

● i18n-ignore-rewrite-source-locale with basepath › get _next/static/ files by skipping locale in rewrite, locale:

ENOENT: no such file or directory, scandir '/tmp/next-install-61b454c77352ba350b3ea59c5cac4798adf87a8429660a2a4001118f5b324cab/.next/static/chunks'

● i18n-ignore-rewrite-source-locale with basepath › get _next/static/ files by skipping locale in rewrite, locale: /en

ENOENT: no such file or directory, scandir '/tmp/next-install-61b454c77352ba350b3ea59c5cac4798adf87a8429660a2a4001118f5b324cab/.next/static/chunks'

● i18n-ignore-rewrite-source-locale with basepath › get _next/static/ files by skipping locale in rewrite, locale: /sv

ENOENT: no such file or directory, scandir '/tmp/next-install-61b454c77352ba350b3ea59c5cac4798adf87a8429660a2a4001118f5b324cab/.next/static/chunks'

● i18n-ignore-rewrite-source-locale with basepath › get _next/static/ files by skipping locale in rewrite, locale: /nl

ENOENT: no such file or directory, scandir '/tmp/next-install-61b454c77352ba350b3ea59c5cac4798adf87a8429660a2a4001118f5b324cab/.next/static/chunks'

test/integration/404-page-app/test/index.test.js

  • 404 Page Support with _app > dev mode > should not show pages/404 GIP error if _app has GIP
Expand output

● 404 Page Support with _app › dev mode › should not show pages/404 GIP error if _app has GIP

expect(received).toBe(expected) // Object.is equality

Expected: 404
Received: 500

  79 |     it('should not show pages/404 GIP error if _app has GIP', async () => {
  80 |       const res = await fetchViaHTTP(appPort, '/abc')
> 81 |       expect(res.status).toBe(404)
     |                          ^
  82 |       const $ = cheerio.load(await res.text())
  83 |       expect($('#404-title').text()).toBe('Hi There')
  84 |       expect(stderr).not.toMatch(gip404Err)

  at Object.<anonymous> (integration/404-page-app/test/index.test.js:81:26)

test/development/acceptance/ReactRefresh.test.ts

  • ReactRefresh > can edit a component without losing state
  • ReactRefresh > cyclic dependencies
Expand output

● ReactRefresh › can edit a component without losing state

Application is in inconsistent state: timeout.

  82 |           }
  83 |           if (status !== 'pending') {
> 84 |             throw new Error(`Application is in inconsistent state: ${status}.`)
     |                   ^
  85 |           }
  86 |
  87 |           await new Promise((resolve) => setTimeout(resolve, 30))

  at Object.patch (development/acceptance/helpers.ts:84:19)
  at Object.<anonymous> (development/acceptance/ReactRefresh.test.ts:20:5)

● ReactRefresh › cyclic dependencies

Application is in inconsistent state: timeout.

  82 |           }
  83 |           if (status !== 'pending') {
> 84 |             throw new Error(`Application is in inconsistent state: ${status}.`)
     |                   ^
  85 |           }
  86 |
  87 |           await new Promise((resolve) => setTimeout(resolve, 30))

  at Object.patch (development/acceptance/helpers.ts:84:19)
  at Object.<anonymous> (development/acceptance/ReactRefresh.test.ts:157:5)

test/e2e/app-dir/app-alias/app-alias.test.ts

  • app-dir alias handling > should handle typescript paths alias correctly
  • app-dir alias handling > should resolve css imports from outside with src folder presented
Expand output

● app-dir alias handling › should handle typescript paths alias correctly

expect(received).toContain(expected) // indexOf

Expected substring: "click</button>"
Received string:    "failed to canonicalize embedded file path·
Caused by:
    No such file or directory (os error 2)"

  14 |     it('should handle typescript paths alias correctly', async () => {
  15 |       const html = await next.render('/button')
> 16 |       expect(html).toContain('click</button>')
     |                    ^
  17 |     })
  18 |
  19 |     it('should resolve css imports from outside with src folder presented', async () => {

  at Object.<anonymous> (e2e/app-dir/app-alias/app-alias.test.ts:16:20)

● app-dir alias handling › should resolve css imports from outside with src folder presented

page.waitForSelector: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator('button')
============================================================

  329 |     return this.chain(() => {
  330 |       return page
> 331 |         .waitForSelector(selector, { timeout, state: 'attached' })
      |          ^
  332 |         .then(async (el) => {
  333 |           // it seems selenium waits longer and tests rely on this behavior
  334 |           // so we wait for the load event fire before returning

  at lib/browsers/playwright.ts:331:10

test/integration/bigint/test/index.test.js

  • bigint API route support > dev mode > should return 200
  • bigint API route support > dev mode > should return the BigInt result text
Expand output

● bigint API route support › dev mode › should return 200

expect(received).toEqual(expected) // deep equality

Expected: 200
Received: 500

  22 |       method: 'GET',
  23 |     })
> 24 |     expect(res.status).toEqual(200)
     |                        ^
  25 |   })
  26 |
  27 |   it('should return the BigInt result text', async () => {

  at Object.<anonymous> (integration/bigint/test/index.test.js:24:24)

● bigint API route support › dev mode › should return the BigInt result text

expect(received).toEqual(expected) // deep equality

Expected: "3"
Received: false

  29 |       method: 'GET',
  30 |     }).then((res) => res.ok && res.text())
> 31 |     expect(resText).toEqual('3')
     |                     ^
  32 |   })
  33 | }
  34 |

  at Object.<anonymous> (integration/bigint/test/index.test.js:31:21)

@kwonoj kwonoj force-pushed the test-results-diff branch from 4ae88f2 to 560a8cb Compare January 31, 2023 16:20
@github-actions
Copy link
Contributor

Benchmark for 582b52a

Click to view benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 9080.79µs ± 63.13µs 9083.03µs ± 86.19µs +0.02%
bench_hmr_to_commit/Turbopack RCC/1000 modules 9409.43µs ± 87.45µs 9425.74µs ± 68.68µs +0.17%
bench_hmr_to_commit/Turbopack RSC/1000 modules 470.57ms ± 2.84ms 473.19ms ± 2.65ms +0.56%
bench_hmr_to_commit/Turbopack SSR/1000 modules 9342.14µs ± 82.01µs 9302.04µs ± 67.94µs -0.43%
bench_hmr_to_eval/Turbopack CSR/1000 modules 8102.42µs ± 42.61µs 8188.38µs ± 74.40µs +1.06%
bench_hmr_to_eval/Turbopack RCC/1000 modules 8357.24µs ± 60.49µs 8307.89µs ± 75.46µs -0.59%
bench_hmr_to_eval/Turbopack SSR/1000 modules 8345.49µs ± 63.75µs 8366.17µs ± 71.83µs +0.25%
bench_hydration/Turbopack RCC/1000 modules 3718.79ms ± 9.56ms 3738.03ms ± 14.19ms +0.52%
bench_hydration/Turbopack RSC/1000 modules 3248.73ms ± 9.49ms 3254.11ms ± 5.95ms +0.17%
bench_hydration/Turbopack SSR/1000 modules 3020.43ms ± 10.50ms 2992.89ms ± 7.26ms -0.91%
bench_startup/Turbopack CSR/1000 modules 2048.38ms ± 6.42ms 2008.49ms ± 16.77ms -1.95%
bench_startup/Turbopack RCC/1000 modules 2462.51ms ± 5.70ms 2454.95ms ± 3.49ms -0.31%
bench_startup/Turbopack RSC/1000 modules 2354.30ms ± 5.35ms 2362.32ms ± 7.38ms +0.34%
bench_startup/Turbopack SSR/1000 modules 2003.95ms ± 3.80ms 2001.92ms ± 3.50ms -0.10%

@kwonoj kwonoj force-pushed the test-results-diff branch 2 times, most recently from 811d4b7 to b566539 Compare January 31, 2023 18:30
@kwonoj kwonoj changed the title [DONOTMERGE] WIP ci(workflow): format integration test results Jan 31, 2023
@kwonoj kwonoj marked this pull request as ready for review January 31, 2023 18:48
@kwonoj kwonoj requested review from a team as code owners January 31, 2023 18:48
@github-actions
Copy link
Contributor

Benchmark for ef6f403

Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 8614.87µs ± 153.11µs 7737.88µs ± 79.33µs -10.18% -4.96%
bench_hmr_to_commit/Turbopack RSC/1000 modules 466.98ms ± 4.32ms 489.80ms ± 5.01ms +4.89% +0.87%
bench_hmr_to_commit/Turbopack SSR/1000 modules 8402.42µs ± 142.31µs 9505.67µs ± 112.90µs +13.13% +6.82%
bench_hmr_to_eval/Turbopack SSR/1000 modules 7842.85µs ± 171.62µs 8443.41µs ± 98.47µs +7.66% +0.74%
bench_hydration/Turbopack RCC/1000 modules 3668.56ms ± 11.54ms 3727.72ms ± 10.73ms +1.61% +0.40%
bench_hydration/Turbopack RSC/1000 modules 3349.82ms ± 8.41ms 3420.77ms ± 20.63ms +2.12% +0.38%
bench_hydration/Turbopack SSR/1000 modules 3154.36ms ± 18.96ms 3011.59ms ± 14.68ms -4.53% -2.42%
bench_startup/Turbopack RCC/1000 modules 2473.10ms ± 6.09ms 2429.29ms ± 6.84ms -1.77% -0.73%
Click to view full benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 8614.87µs ± 153.11µs 7737.88µs ± 79.33µs -10.18% -4.96%
bench_hmr_to_commit/Turbopack RCC/1000 modules 8898.86µs ± 191.81µs 8774.27µs ± 99.94µs -1.40%
bench_hmr_to_commit/Turbopack RSC/1000 modules 466.98ms ± 4.32ms 489.80ms ± 5.01ms +4.89% +0.87%
bench_hmr_to_commit/Turbopack SSR/1000 modules 8402.42µs ± 142.31µs 9505.67µs ± 112.90µs +13.13% +6.82%
bench_hmr_to_eval/Turbopack CSR/1000 modules 8151.82µs ± 56.13µs 7967.90µs ± 84.31µs -2.26%
bench_hmr_to_eval/Turbopack RCC/1000 modules 7416.47µs ± 112.73µs 7368.69µs ± 95.13µs -0.64%
bench_hmr_to_eval/Turbopack SSR/1000 modules 7842.85µs ± 171.62µs 8443.41µs ± 98.47µs +7.66% +0.74%
bench_hydration/Turbopack RCC/1000 modules 3668.56ms ± 11.54ms 3727.72ms ± 10.73ms +1.61% +0.40%
bench_hydration/Turbopack RSC/1000 modules 3349.82ms ± 8.41ms 3420.77ms ± 20.63ms +2.12% +0.38%
bench_hydration/Turbopack SSR/1000 modules 3154.36ms ± 18.96ms 3011.59ms ± 14.68ms -4.53% -2.42%
bench_startup/Turbopack CSR/1000 modules 2162.70ms ± 19.17ms 2161.80ms ± 11.71ms -0.04%
bench_startup/Turbopack RCC/1000 modules 2473.10ms ± 6.09ms 2429.29ms ± 6.84ms -1.77% -0.73%
bench_startup/Turbopack RSC/1000 modules 2324.74ms ± 10.36ms 2345.47ms ± 6.50ms +0.89%
bench_startup/Turbopack SSR/1000 modules 1990.15ms ± 7.69ms 1964.93ms ± 5.20ms -1.27%

@kwonoj kwonoj added the pr: automerge Kodiak will merge these automatically after checks pass label Jan 31, 2023
@github-actions
Copy link
Contributor

Benchmark for d6b21e2

Click to view benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 8901.38µs ± 62.93µs 8938.85µs ± 61.26µs +0.42%
bench_hmr_to_commit/Turbopack RCC/1000 modules 9268.54µs ± 84.48µs 9249.52µs ± 55.88µs -0.21%
bench_hmr_to_commit/Turbopack RSC/1000 modules 480.78ms ± 2.22ms 475.76ms ± 1.68ms -1.04%
bench_hmr_to_commit/Turbopack SSR/1000 modules 9194.34µs ± 47.30µs 9139.58µs ± 47.77µs -0.60%
bench_hmr_to_eval/Turbopack CSR/1000 modules 8055.50µs ± 89.01µs 7961.71µs ± 88.17µs -1.16%
bench_hmr_to_eval/Turbopack RCC/1000 modules 8184.10µs ± 81.99µs 8161.50µs ± 49.34µs -0.28%
bench_hmr_to_eval/Turbopack SSR/1000 modules 8239.29µs ± 70.08µs 8171.41µs ± 72.16µs -0.82%
bench_hydration/Turbopack RCC/1000 modules 3694.07ms ± 8.00ms 3682.16ms ± 8.95ms -0.32%
bench_hydration/Turbopack RSC/1000 modules 3233.73ms ± 12.93ms 3222.42ms ± 11.83ms -0.35%
bench_hydration/Turbopack SSR/1000 modules 2977.78ms ± 8.29ms 2987.12ms ± 8.67ms +0.31%
bench_startup/Turbopack CSR/1000 modules 2022.00ms ± 11.01ms 2045.45ms ± 17.03ms +1.16%
bench_startup/Turbopack RCC/1000 modules 2431.94ms ± 5.50ms 2435.94ms ± 6.57ms +0.16%
bench_startup/Turbopack RSC/1000 modules 2346.64ms ± 7.02ms 2338.99ms ± 2.60ms -0.33%
bench_startup/Turbopack SSR/1000 modules 1989.04ms ± 6.16ms 1993.17ms ± 4.61ms +0.21%

@kwonoj kwonoj merged commit ad11075 into main Feb 1, 2023
@kwonoj kwonoj deleted the test-results-diff branch February 1, 2023 01:24
mehulkar pushed a commit that referenced this pull request Feb 1, 2023
Closes WEB-486.

This PR amends stat report action formatting - mainly introducing diff
summary between base (`main` for the PR, `release` to `release` for new
next.js release). This summary will be piped into slack channel once
this PR lands.

There maybe some things to amend further, but primary logic seems to
working. Please note current base (`main`) results are incorrectly
collected, makes diff summary looks weird. We may need wipe out
incorrect data.
kwonoj added a commit that referenced this pull request Feb 1, 2023
Requires #3534

Partially progresses WEB-460

This PR creates a plain text summary, so subsequent workflow can
download it via download-artifact then broadcast as needed. We'll use it
to posting into Slack channel. Action will only generate text file when
we supposed to report (currently, release-to-release) so if
download-artifact does not contain text-summary.md, it can skip to
broadcast into Slack channel.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: ci pr: automerge Kodiak will merge these automatically after checks pass
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants