[devicelab] Refresh documentation #80064

CaseyHillers · 2021-04-08T17:15:48Z

Issues

Pre-launch Checklist

I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
I read the [Tree Hygiene] wiki page, which explains my responsibilities.
I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement].
I signed the [CLA].
I listed at least one issue that this PR fixes in the description above.
I updated/added relevant documentation (doc comments with ///).
I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt.
All existing and new tests are passing.

CaseyHillers · 2021-04-08T17:16:35Z

dev/devicelab/README.md

-
-This defaults to only running tests supported by your host device's platform
-(`--match-host-platform`) and exiting after the first failure (`--exit`).
+`TODO(chillers): Update to use .ci.yaml. https://github.com/flutter/flutter/issues/74660`

 ### Running specific tests


test_runner.dart currently does not support this. Should this be ported over or removed?

CaseyHillers · 2021-04-08T17:18:22Z

dev/devicelab/README.md

+1. Add prod builder to [flutter/infra devicelab_config.star](https://github.com/flutter/infra/blob/master/config/devicelab_config.star)
+  - Example PR: https://github.com/flutter/infra/pull/401/files
+  - This will need to soak for 15 minutes after merged to propagate
+  - There are various lists for the different testbeds a test can run on


In flutter/cocoon, I plan to add a more robust testbed document listing what there is in the devicelab after we move from the luci-scheduler to the cocoon-scheduler

What are these various lists?

linux_tasks, linux_vm_tasks, mac_android_tasks, and mac_ios_tasks. I'd to formalize these testbeds in cocoon when we start using the cocoon-scheduler

jmagman · 2021-04-08T17:24:31Z

dev/devicelab/README.md

 for information on using the dashboards.

-## How the DeviceLab runs tasks
+## Table of Contents
+* [How the DeviceLab runs tests](#how-the-devicelab-runs-tasks)


#how-the-devicelab-runs-tests

Nice catch!

jmagman · 2021-04-08T17:25:50Z

dev/devicelab/README.md

-
-This defaults to only running tests supported by your host device's platform
-(`--match-host-platform`) and exiting after the first failure (`--exit`).
+`TODO(chillers): Update to use .ci.yaml. https://github.com/flutter/flutter/issues/74660`


May as well leave off this section until it's done, or you'll get pinged about it in 10 years.

Thanks, removed

dev/devicelab/README.md

jmagman · 2021-04-08T17:30:16Z

dev/devicelab/README.md

+
+1. Add prod builder to [flutter/infra devicelab_config.star](https://github.com/flutter/infra/blob/master/config/devicelab_config.star)
+  - Example PR: https://github.com/flutter/infra/pull/401/files
+  - This will need to soak for 15 minutes after merged to propagate


Maybe instructions for how you can tell it's ready?

Done - Check LUCI console for devicelab tasks

jmagman · 2021-04-08T17:34:15Z

dev/devicelab/README.md

+
+Flutter's DeviceLab does not currently have capacity to run tests against physical devices in presubmit.
+
+Host only tests are okay. First add them to [continuous-integration](#adding-tests-to-ci), and after marked blocking add them to presubmit.


Whoa, I didn't know this, neat.

In general I think we want to discourage these kinds of tests except in rare situations. See #65790.

Maybe we can encourage people to consider adding a flutter_tools test instead?

If that's the approach, i'll give the advice of adding them as flutter_tools tests. Is there documentation I should refer to?

No, I think there are good use cases for adding host tests the devicelab, (for example the devicelab machines have the Apple development provisioning profiles set up, so #77428 for example can't been a flutter_tools test), and maybe some more desktop test reasons.

Maybe what you had before, but with a caveat like:

Also consider creating your integration test in packages/flutter_tools/test/integration.shard to avoid using limited DeviceLab scheduling resources. Example: https://github.com/flutter/flutter/pull/73577/files"

Great! Added

dev/devicelab/README.md

keyonghan · 2021-04-08T18:59:04Z

dev/devicelab/README.md

+1. Add prod builder to [flutter/infra devicelab_config.star](https://github.com/flutter/infra/blob/master/config/devicelab_config.star)
+  - Example PR: https://github.com/flutter/infra/pull/401/files
+  - This will need to soak for 15 minutes after merged to propagate
+  - There are various lists for the different testbeds a test can run on


What are these various lists?

dev/devicelab/README.md

keyonghan

LGTM with nit.
Leave approval to jmagman@

dev/devicelab/README.md

jmagman · 2021-04-08T21:20:26Z

dev/devicelab/README.md

+
+## Adding tests to presubmit
+
+Flutter's DeviceLab does not currently have capacity to run tests against physical devices in presubmit.


I liked your previous instructions to add to presubmit, I'm going to use them now that I know this feature exists!

jmagman · 2021-04-09T17:24:48Z

dev/devicelab/README.md

-    stage: {STAGE}
-    required_agent_capabilities: {CAPABILITIES}
-```
+Host only tests should be added to `flutter_tools`.


It sounds like the following instructions are to add it to flutter_tools but the rest of this section is how to not add it to flutter_tools...

How about removing this sentence and add at the bottom of the section something like:
"Note that DeviceLab tests should generally require a tethered device. If you are adding host-only tests, considering adding your test to packages/flutter_tools/test/integration.shard ."

Thanks, updated.

dev/devicelab/README.md

CaseyHillers · 2021-04-14T16:33:05Z

jmagman

LGTM, thanks @CaseyHillers!

[devicelab] Refresh documentation

58346ca

flutter-dashboard bot added the c: contributor-productivity Team-specific productivity, code health, technical debt. label Apr 8, 2021

google-cla bot added the cla: yes label Apr 8, 2021

CaseyHillers commented Apr 8, 2021

View reviewed changes

Update README.md

36e47e2

CaseyHillers commented Apr 8, 2021

View reviewed changes

CaseyHillers requested review from keyonghan, christopherfujino and jmagman April 8, 2021 17:18

jmagman reviewed Apr 8, 2021

View reviewed changes

Review comments

71a0799

keyonghan reviewed Apr 8, 2021

View reviewed changes

christopherfujino reviewed Apr 8, 2021

View reviewed changes

dev/devicelab/README.md Outdated Show resolved Hide resolved

keyonghan reviewed Apr 8, 2021

View reviewed changes

Review changes - remove anchor links, rewording

245b37a

keyonghan reviewed Apr 8, 2021

View reviewed changes

dev/devicelab/README.md Outdated Show resolved Hide resolved

jmagman reviewed Apr 8, 2021

View reviewed changes

Update README.md

f93c909

jmagman reviewed Apr 9, 2021

View reviewed changes

dev/devicelab/README.md Outdated Show resolved Hide resolved

fix presubmit instructions

71dabb4

jmagman approved these changes Apr 14, 2021

View reviewed changes

CaseyHillers added waiting for tree to go green team-infra Owned by Infrastructure team labels Apr 14, 2021

fluttergithubbot merged commit d55b8c0 into master Apr 14, 2021

fluttergithubbot deleted the CaseyHillers-patch-1 branch April 14, 2021 20:49


		Flutter's DeviceLab does not currently have capacity to run tests against physical devices in presubmit.

		Host only tests are okay. First add them to [continuous-integration](#adding-tests-to-ci), and after marked blocking add them to presubmit.


		## Adding tests to presubmit

		Flutter's DeviceLab does not currently have capacity to run tests against physical devices in presubmit.

[devicelab] Refresh documentation #80064

[devicelab] Refresh documentation #80064

Uh oh!

Conversation

CaseyHillers commented Apr 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issues

Pre-launch Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

keyonghan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CaseyHillers commented Apr 14, 2021

Uh oh!

jmagman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CaseyHillers commented Apr 8, 2021 •

edited

Loading