这是indexloc提供的服务，不要输入任何密码
Skip to content

Randomize tests, exclude tests that fail with randomization. #86793

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jul 27, 2021
Merged

Randomize tests, exclude tests that fail with randomization. #86793

merged 2 commits into from
Jul 27, 2021

Conversation

gspencergoog
Copy link
Contributor

@gspencergoog gspencergoog commented Jul 21, 2021 •
edited
Loading

Description

This re-lands PR #85159, now that the test package has been published with support for turning off randomization for a particular tag.

This change will cause tests to be run in a random order, determined by a random seed that is set to YYYYMMDD, so that the same test will continue to fail on the same day. The seed changes at 7am UTC. To reproduce the failure, developers can run locally with the same seed as the failure (which is printed in the test logs).

To determine which tests fail with it on, I ran all the tests 100 times with different random shuffle seeds, and then also ran it with the date seeds from today until the end of August, and tagged all of the test suites (files) that fail, with a seed that caused them to fail.

Related Issues

Tests

@flutter-dashboard flutter-dashboard bot added f: cupertino flutter/packages/flutter/cupertino repository f: material design flutter/packages/flutter/material repository. framework flutter/packages/flutter repository. See also f: labels. c: contributor-productivity Team-specific productivity, code health, technical debt. labels Jul 21, 2021
@google-cla google-cla bot added the cla: yes label Jul 21, 2021
@gspencergoog gspencergoog marked this pull request as draft July 21, 2021 20:45
@flutter-dashboard
Copy link

This pull request has been changed to a draft. The currently pending flutter-gold status will not be able to resolve until a new commit is pushed or the change is marked ready for review again.

For more guidance, visit Writing a golden file test for package:flutter.

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

@gspencergoog gspencergoog force-pushed the randomize_tests branch 3 times, most recently from 5dccbfe to ab1e4b0 Compare July 22, 2021 17:01
@gspencergoog gspencergoog marked this pull request as ready for review July 22, 2021 17:02
@gspencergoog gspencergoog requested a review from Piinks as a code owner July 22, 2021 17:02
@flutter-dashboard flutter-dashboard bot added the a: tests "flutter test", flutter_test, or one of our tests label Jul 22, 2021
@gspencergoog gspencergoog force-pushed the randomize_tests branch 3 times, most recently from ca69209 to 0d7eb05 Compare July 23, 2021 17:55
@gspencergoog gspencergoog mentioned this pull request Jul 23, 2021
Open
@flutter-dashboard flutter-dashboard bot added the tool Affects the "flutter" command-line tool. See also t: labels. label Jul 23, 2021
darrenaustin
darrenaustin approved these changes Jul 27, 2021
View reviewed changes
Copy link
Contributor

@darrenaustin darrenaustin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice work.

dev/bots/test.dart
String? _shuffleSeed;
String get shuffleSeed {
if (_shuffleSeed == null) {
// Change the seed at 7am, UTC.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 7am UTC? Just curious.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is approximately midnight Pacific, and for the majority of Flutter framework developers will switch sometime between midnight and 9am. I would have made it be midnight Pacific, except that Dart has no way to compute whether it is daylight savings time or not, so it is sometimes 11pm, sometimes midnight. I also didn't want to make it midnight local time, since who knows where our servers will be running the tests in the future, so UTC was a better bet.

@darrenaustin darrenaustin merged commit 738ce43 into flutter:master Jul 27, 2021
@jmagman jmagman mentioned this pull request Aug 18, 2021
Closed
@gspencergoog gspencergoog deleted the randomize_tests branch October 7, 2022 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@darrenaustin darrenaustin darrenaustin approved these changes

@Piinks Piinks Awaiting requested review from Piinks

Assignees
No one assigned
Labels
a: tests "flutter test", flutter_test, or one of our tests c: contributor-productivity Team-specific productivity, code health, technical debt. f: cupertino flutter/packages/flutter/cupertino repository f: material design flutter/packages/flutter/material repository. framework flutter/packages/flutter repository. See also f: labels. tool Affects the "flutter" command-line tool. See also t: labels.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Start running tests with --test-randomize-ordering-seed=random
2 participants
@gspencergoog @darrenaustin