feat: tests can opt-in to retries #754
Conversation
ab141d6 to
f1a9b69
Compare
Yes, I think I'm up for having this added.
Yes I think it probably should be exposed. Historically I think (speculating here) the reason zero context has been given is just because we're clearing out the environment and isolating tests and exposing any context would mean adding a new API surface to explicitly start making some things available inside the isolated test runner scope. |
|
Thanks, I'll follow up on this then. Any guidance on the interface? Defining new magic globals like |
|
That sounds okay, but the code you used is a little confusing. We should either set the number of retries (zero being don't attempt any retries after failure) and default it to zero or set the max attempts and default it to one, but preferably not mix and match and have to calculate one from the other in code. Not that the math is hard, just that the code will be neater if the logic aligns with the interface naming the end user is seeing. |
|
Don't look at the code yet, it's a broken mess :) |
f1a9b69 to
16b5e8a
Compare
| assert(type(match) == 'table') | ||
| assert(type(assert) == 'table') | ||
|
|
||
| describe('retry test', function() |
There was a problem hiding this comment.
should the set_retries tests go in a new _spec.lua file?
| if status == 'success' then | ||
| -- Remove any failures for this test since it succeeded | ||
| for i = #handler.failures, 1, -1 do | ||
| if handler.failures[i].id == tostring(element) then | ||
| table.remove(handler.failures, i) |
There was a problem hiding this comment.
This feels a bit hacky. Is there a better way for the it() implementation in busted/init.lua to avoid reporting "failures" until all retries are exhausted?
16b5e8a to
4fe1157
Compare
Problem:
Tests have no way to retry without manually managing `before_each` and
`after_each` setup/teardown. This is painful. And busted does a good job
of guarding its internals and makes it impossible to extend it: AFAICT,
consumers have no way to access the current "test context", or the
current list of `before_each`/`after_each` hooks.
Solution:
Introduce `env.set_retries()`, which allows test authors to optionally retry a test:
```lua
it('...', function()
set_retries(2) -- If the test fails, it will be retried up to 2 times.
end)
```
Testing:
luarocks --local remove busted --force && luarocks --local make && ~/.luarocks/bin/busted --pattern=core
4fe1157 to
bf3a5dd
Compare
|
Still needs cleanup (and more tests), but the functionality (AI-generated) seems to work now. Note: On a fresh clone of this repo there are |
Problem
Tests have no way to retry without manually managing
before_eachandafter_eachsetup/teardown. This is painful. And busted does a good job of guarding its internals and makes it impossible to extend it: AFAICT, consumers have no way to access the current "test context", or the current list ofbefore_each/after_eachhooks.Solution
Introduce
env.set_retries(), which allows test authors to optionally retry a test:Todo
91 failures(and 1583 successes).Haven't added docs/tests, want to see if this will be acceptable first.