+
Skip to content

[RFC] tests: add test_todo() for known failures #1374

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[RFC] tests: add test_todo() for known failures #1374

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
472d051
[RFC] tests: add test_todo() to mark known breakages
phillipwood Oct 2, 2022
645fa29
[RFC] test_todo: allow [!] grep as the command
phillipwood Oct 2, 2022
c0d955c
[RFC] test_todo: allow [verbose] test as the command
phillipwood Oct 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions t/README
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -892,6 +892,10 @@ see test-lib-functions.sh for the full list and their options.

- test_expect_failure [<prereq>] <message> <script>

Where possible new tests should use test_expect_success and mark
the individual failing commands with test_todo (see below) rather
than using test_expect_failure.

This is NOT the opposite of test_expect_success, but is used
to mark a test that demonstrates a known breakage. Unlike
the usual test_expect_success tests, which say "ok" on
Expand All @@ -902,6 +906,14 @@ see test-lib-functions.sh for the full list and their options.
Like test_expect_success this function can optionally use a three
argument invocation with a prerequisite as the first argument.

- test_todo <command>

This is used to mark commands that should succeed but do not due to
a known issue. The test will be reported as a known failure if the
issue still exists and won’t cause -i (immediate) to stop. If the
command unexpectedly succeeds then the test will be reported as a
failing. test_todo cannot be used in a subshell.

- test_debug <script>

This takes a single argument, <script>, and evaluates it only
Expand Down
92 changes: 92 additions & 0 deletions t/t0000-basic.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,98 @@ test_expect_success 'subtest: a passing TODO test' '
EOF
'

test_expect_success 'subtest: a failing test_todo' '
write_and_run_sub_test_lib_test failing-test-todo <<-\EOF &&
test_false () {
false
}
test_expect_success "passing test" "true"
test_expect_success "known todo" "test_todo test_false"
test_done
EOF
check_sub_test_lib_test failing-test-todo <<-\EOF
> ok 1 - passing test
> not ok 2 - known todo # TODO known breakage
> # still have 1 known breakage(s)
> # passed all remaining 1 test(s)
> 1..2
EOF
'

test_expect_success 'subtest: a passing test_todo' '
write_and_run_sub_test_lib_test_err passing-test-todo <<-\EOF &&
test_true () {
true
}
test_expect_success "pretend we have fixed a test_todo breakage" \
"test_todo test_true"
test_done
EOF
check_sub_test_lib_test passing-test-todo <<-\EOF
> not ok 1 - pretend we have fixed a test_todo breakage
> # test_todo test_true
> # failed 1 among 1 test(s)
> 1..1
EOF
'

test_expect_success 'subtest: test_todo allowed arguments' '
write_and_run_sub_test_lib_test_err acceptable-test-todo <<-\EOF &&
# This an acceptable command for test_todo but not test_must_fail
test_true () {
return 0
}
test_expect_success "test_todo skips env and accepts good command" \
"test_todo env Var=Value git --invalid-option"
test_expect_success "test_todo skips env and rejects bad command" \
"test_todo env Var=Value false"
test_expect_success "test_todo test_must_fail accepts good command" \
"test_todo test_must_fail git --version"
test_expect_success "test_todo test_must_fail rejects bad command" \
"test_todo test_must_fail test_true"
test_expect_success "test_todo accepts grep" \
"echo a >a && test_todo grep b <a"
test_expect_success "test_todo accepts ! grep" \
"echo a >a && test_todo ! grep -v b <a"
test_expect_success "test_todo detects grep errors" \
"echo a >a && test_todo grep --invalid-option b <a"
test_expect_success "test_todo detects ! grep errors" \
"echo a >a && test_todo ! grep --invalid-option -v b <a"
test_expect_success "test_todo accepts test" \
"test_todo test -z a"
test_expect_success "test_todo detects test errors" \
"test_todo test a xxx b"
test_expect_success "test_todo skips verbose and accepts good command" \
"test_todo verbose test -z a"
test_expect_success "test_todo skips verbose and rejects bad command" \
"test_todo verbose false"
test_done
EOF
check_sub_test_lib_test acceptable-test-todo <<-\EOF
> not ok 1 - test_todo skips env and accepts good command # TODO known breakage
> not ok 2 - test_todo skips env and rejects bad command
> # test_todo env Var=Value false
> not ok 3 - test_todo test_must_fail accepts good command # TODO known breakage
> not ok 4 - test_todo test_must_fail rejects bad command
> # test_todo test_must_fail test_true
> not ok 5 - test_todo accepts grep # TODO known breakage
> not ok 6 - test_todo accepts ! grep # TODO known breakage
> not ok 7 - test_todo detects grep errors
> # echo a >a && test_todo grep --invalid-option b <a
> not ok 8 - test_todo detects ! grep errors
> # echo a >a && test_todo ! grep --invalid-option -v b <a
> not ok 9 - test_todo accepts test # TODO known breakage
> not ok 10 - test_todo detects test errors
> # test_todo test a xxx b
> not ok 11 - test_todo skips verbose and accepts good command # TODO known breakage
> not ok 12 - test_todo skips verbose and rejects bad command
> # test_todo verbose false
> # still have 6 known breakage(s)
> # failed 6 among remaining 6 test(s)
> 1..12
EOF
'

test_expect_success 'subtest: 2 TODO tests, one passin' '
write_and_run_sub_test_lib_test partially-passing-todos <<-\EOF &&
test_expect_failure "pretend we have a known breakage" "false"
Expand Down
4 changes: 2 additions & 2 deletions t/t0050-filesystem.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,15 +100,15 @@ test_expect_success CASE_INSENSITIVE_FS 'add directory (with different case)' '
test_cmp expected actual
'

test_expect_failure CASE_INSENSITIVE_FS 'add (with different case)' '
test_expect_success CASE_INSENSITIVE_FS 'add (with different case)' '
git reset --hard initial &&
rm camelcase &&
echo 1 >CamelCase &&
git add CamelCase &&
git ls-files >tmp &&
camel=$(grep -i camelcase tmp) &&
test $(echo "$camel" | wc -l) = 1 &&
test "z$(git cat-file blob :$camel)" = z1
test_todo test "z$(git cat-file blob :$camel)" = z1
'

test_expect_success "setup unicode normalization tests" '
Expand Down
12 changes: 6 additions & 6 deletions t/t3401-rebase-and-am-rename.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ test_expect_success 'rebase --interactive: directory rename detected' '
)
'

test_expect_failure 'rebase --apply: directory rename detected' '
test_expect_success 'rebase --apply: directory rename detected' '
(
cd dir-rename &&

Expand All @@ -63,8 +63,8 @@ test_expect_failure 'rebase --apply: directory rename detected' '
git ls-files -s >out &&
test_line_count = 5 out &&

test_path_is_file y/d &&
test_path_is_missing x/d
test_todo test_path_is_file y/d &&
test_todo test_path_is_missing x/d
)
'

Expand All @@ -84,7 +84,7 @@ test_expect_success 'rebase --merge: directory rename detected' '
)
'

test_expect_failure 'am: directory rename detected' '
test_expect_success 'am: directory rename detected' '
(
cd dir-rename &&

Expand All @@ -97,8 +97,8 @@ test_expect_failure 'am: directory rename detected' '
git ls-files -s >out &&
test_line_count = 5 out &&

test_path_is_file y/d &&
test_path_is_missing x/d
test_todo test_path_is_file y/d &&
test_todo test_path_is_missing x/d
)
'

Expand Down
6 changes: 3 additions & 3 deletions t/t3424-rebase-empty.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,15 +34,15 @@ test_expect_success 'setup test repository' '
git commit -m "Five letters ought to be enough for anybody"
'

test_expect_failure 'rebase (apply-backend)' '
test_when_finished "git rebase --abort" &&
test_expect_success 'rebase (apply-backend)' '
test_when_finished "test_might_fail git rebase --abort" &&
git checkout -B testing localmods &&
# rebase (--apply) should not drop commits that start empty
git rebase --apply upstream &&

test_write_lines D C B A >expect &&
git log --format=%s >actual &&
test_cmp expect actual
test_todo test_cmp expect actual
'

test_expect_success 'rebase --merge --empty=drop' '
Expand Down
12 changes: 6 additions & 6 deletions t/t3510-cherry-pick-sequence.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -577,7 +577,7 @@ test_expect_success '--continue respects -x in first commit in multi-pick' '
grep "cherry picked from.*$picked" msg
'

test_expect_failure '--signoff is automatically propagated to resolved conflict' '
test_expect_success '--signoff is automatically propagated to resolved conflict' '
pristine_detach initial &&
test_expect_code 1 git cherry-pick --signoff base..anotherpick &&
echo "c" >foo &&
Expand All @@ -591,11 +591,11 @@ test_expect_failure '--signoff is automatically propagated to resolved conflict'
git cat-file commit HEAD~3 >initial_msg &&
! grep "Signed-off-by:" initial_msg &&
grep "Signed-off-by:" unrelatedpick_msg &&
! grep "Signed-off-by:" picked_msg &&
test_todo ! grep "Signed-off-by:" picked_msg &&
grep "Signed-off-by:" anotherpick_msg
'

test_expect_failure '--signoff dropped for implicit commit of resolution, multi-pick case' '
test_expect_success '--signoff dropped for implicit commit of resolution, multi-pick case' '
pristine_detach initial &&
test_must_fail git cherry-pick -s picked anotherpick &&
echo c >foo &&
Expand All @@ -605,10 +605,10 @@ test_expect_failure '--signoff dropped for implicit commit of resolution, multi-
git diff --exit-code HEAD &&
test_cmp_rev initial HEAD^^ &&
git cat-file commit HEAD^ >msg &&
! grep Signed-off-by: msg
test_todo ! grep Signed-off-by: msg
'

test_expect_failure 'sign-off needs to be reaffirmed after conflict resolution, single-pick case' '
test_expect_success 'sign-off needs to be reaffirmed after conflict resolution, single-pick case' '
pristine_detach initial &&
test_must_fail git cherry-pick -s picked &&
echo c >foo &&
Expand All @@ -618,7 +618,7 @@ test_expect_failure 'sign-off needs to be reaffirmed after conflict resolution,
git diff --exit-code HEAD &&
test_cmp_rev initial HEAD^ &&
git cat-file commit HEAD >msg &&
! grep Signed-off-by: msg
test_todo ! grep Signed-off-by: msg
'

test_expect_success 'malformed instruction sheet 1' '
Expand Down
8 changes: 4 additions & 4 deletions t/t3600-rm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -790,18 +790,18 @@ test_expect_success SYMLINKS 'rm across a symlinked leading path (no index)' '
test_path_is_file e/f
'

test_expect_failure SYMLINKS 'rm across a symlinked leading path (w/ index)' '
test_expect_success SYMLINKS 'rm across a symlinked leading path (w/ index)' '
rm -rf d e &&
mkdir d &&
echo content >d/f &&
git add -A e d &&
git commit -m "d/f exists" &&
mv d e &&
ln -s e d &&
test_must_fail git rm d/f &&
git rev-parse --verify :d/f &&
test_todo test_must_fail git rm d/f &&
test_todo git rev-parse --verify :d/f &&
test -h d &&
test_path_is_file e/f
test_todo test_path_is_file e/f
'

test_expect_success 'setup for testing rm messages' '
Expand Down
12 changes: 6 additions & 6 deletions t/t3903-stash.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -556,7 +556,7 @@ test_expect_success 'unstash must re-create the file' '
test bar = "$(cat file)"
'

test_expect_failure 'stash directory to file' '
test_expect_success 'stash directory to file' '
git reset --hard &&
mkdir dir &&
echo foo >dir/file &&
Expand All @@ -567,12 +567,12 @@ test_expect_failure 'stash directory to file' '
git stash save "directory to file" &&
test_path_is_dir dir &&
test foo = "$(cat dir/file)" &&
test_must_fail git stash apply &&
test bar = "$(cat dir)" &&
test_todo test_must_fail git stash apply &&
test_todo test bar = "$(cat dir)" &&
git reset --soft HEAD^
'

test_expect_failure 'stash file to directory' '
test_expect_success 'stash file to directory' '
git reset --hard &&
rm file &&
mkdir file &&
Expand All @@ -581,8 +581,8 @@ test_expect_failure 'stash file to directory' '
test_path_is_file file &&
test bar = "$(cat file)" &&
git stash apply &&
test_path_is_file file/file &&
test foo = "$(cat file/file)"
test_todo test_path_is_file file/file &&
test_todo test foo = "$(cat file/file)"
'

test_expect_success 'giving too many ref arguments does not modify files' '
Expand Down
20 changes: 10 additions & 10 deletions t/t4014-format-patch.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,12 +165,12 @@ test_expect_success 'additional command line cc (ascii)' '
grep "^ *S E Cipient <scipient@example.com>\$" hdrs5
'

test_expect_failure 'additional command line cc (rfc822)' '
test_expect_success 'additional command line cc (rfc822)' '
git config --replace-all format.headers "Cc: R E Cipient <rcipient@example.com>" &&
git format-patch --cc="S. E. Cipient <scipient@example.com>" --stdout main..side >patch5 &&
sed -e "/^\$/q" patch5 >hdrs5 &&
grep "^Cc: R E Cipient <rcipient@example.com>,\$" hdrs5 &&
grep "^ *\"S. E. Cipient\" <scipient@example.com>\$" hdrs5
test_todo grep "^ *\"S. E. Cipient\" <scipient@example.com>\$" hdrs5
'

test_expect_success 'command line headers' '
Expand All @@ -195,16 +195,16 @@ test_expect_success 'command line To: header (ascii)' '
grep "^To: R E Cipient <rcipient@example.com>\$" hdrs8
'

test_expect_failure 'command line To: header (rfc822)' '
test_expect_success 'command line To: header (rfc822)' '
git format-patch --to="R. E. Cipient <rcipient@example.com>" --stdout main..side >patch8 &&
sed -e "/^\$/q" patch8 >hdrs8 &&
grep "^To: \"R. E. Cipient\" <rcipient@example.com>\$" hdrs8
test_todo grep "^To: \"R. E. Cipient\" <rcipient@example.com>\$" hdrs8
'

test_expect_failure 'command line To: header (rfc2047)' '
test_expect_success 'command line To: header (rfc2047)' '
git format-patch --to="R Ä Cipient <rcipient@example.com>" --stdout main..side >patch8 &&
sed -e "/^\$/q" patch8 >hdrs8 &&
grep "^To: =?UTF-8?q?R=20=C3=84=20Cipient?= <rcipient@example.com>\$" hdrs8
test_todo grep "^To: =?UTF-8?q?R=20=C3=84=20Cipient?= <rcipient@example.com>\$" hdrs8
'

test_expect_success 'configuration To: header (ascii)' '
Expand All @@ -214,18 +214,18 @@ test_expect_success 'configuration To: header (ascii)' '
grep "^To: R E Cipient <rcipient@example.com>\$" hdrs9
'

test_expect_failure 'configuration To: header (rfc822)' '
test_expect_success 'configuration To: header (rfc822)' '
git config format.to "R. E. Cipient <rcipient@example.com>" &&
git format-patch --stdout main..side >patch9 &&
sed -e "/^\$/q" patch9 >hdrs9 &&
grep "^To: \"R. E. Cipient\" <rcipient@example.com>\$" hdrs9
test_todo grep "^To: \"R. E. Cipient\" <rcipient@example.com>\$" hdrs9
'

test_expect_failure 'configuration To: header (rfc2047)' '
test_expect_success 'configuration To: header (rfc2047)' '
git config format.to "R Ä Cipient <rcipient@example.com>" &&
git format-patch --stdout main..side >patch9 &&
sed -e "/^\$/q" patch9 >hdrs9 &&
grep "^To: =?UTF-8?q?R=20=C3=84=20Cipient?= <rcipient@example.com>\$" hdrs9
test_todo grep "^To: =?UTF-8?q?R=20=C3=84=20Cipient?= <rcipient@example.com>\$" hdrs9
'

# check_patch <patch>: Verify that <patch> looks like a half-sane
Expand Down
Loading
点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载