Urbit Testing Procedures #70
Conversation
|
The governance process for |
|
I divided the doc that this UIP previously linked to in two. The "Arvo Testing Standards" section is now contained in the body of the UIP as the formal proposal, and the "Vere" and "Evolving Practices" sections are a single Google doc centered around evolving practices. |
|
Some thoughts on this:
|
There was a problem hiding this comment.
• where should the doc be located?
/tests/readme.md could be good as long for arvo testing instructions/expectations/discipline.
• is the governance process by which the doc gets updated extrajudicial to the UIP process?
If the doc lives in the repo it's gonna have to pass PR review at the very least. In that light, too, seems appropriate to put the doc alongside the tests in the repo where they belong. Local codeowners should generally be responsible for it, but the UIP Shadow Government would do good to keep an eye on things to make sure that's not falling out of line.
| - Catching regressions that may reintroduce known vulnerabilities | ||
| - Identifying memory and runtime issues earlier in development | ||
| - Ensuring correctness of serialization, cryptographic operations, and system boundaries | ||
| - Providing a foundation for automated verification in CI |
There was a problem hiding this comment.
CI is its own whole situation. It's not obvious to me that any of the recommendations made in this doc do anything to improve that. At most it will give you more tests to potentially run during CI.
There was a problem hiding this comment.
Which provides a foundation, right? :)
There was a problem hiding this comment.
I mean, we already have that foundation. "Having tests" doesn't answer anything about:
- How to run the tests automatically,
- Deciding which tests to run automatically,
- Parsing or otherwise retrieving test output for status reports,
and maybe other things I'm not thinking of. I guess my beef here is just that CI seems very out of scope for this document, and so it shouldn't try to talk about it.
Consider this a pre-draft UIP ready for criticism and potential revisions.
A couple key points to resolve before moving to Review: