这是indexloc提供的服务,不要输入任何密码
Skip to content

Conversation

@jcoben
Copy link

@jcoben jcoben commented Oct 19, 2023

What is the purpose of this PR

  • This PR fixes the error resulting from the flaky test: com.insightfullogic.java8.examples.chapter3.RefactorTest#allStringJoins
  • The mentioned test may fail or pass without changes made to the source code when it is run in different JVMs due to HashSet's non-deterministic iteration order.
  • The NonDex tool detects possible issues with "under-determined Java APIs." This means that this problem may come up if in the future someone tries to run the code in a different environment (JVM version, operating system, etc.). Fixing the test now will prevent developers from spending time on this in the future.

Why the tests fail

  • This test fails because the test requests a set of strings returned from code in the Refactor class. This code returns a HashSet containing the string data. The test compares this result to a hard-coded string value. The problem is that the order of iteration is not guaranteed for HashSets. So, the test may fail as the given order can be different from the expected order.

Reproduce the test failure

  • Run the tests with NonDex maven plugin. The commands to recreate the flaky test failures are:
    mvn edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=com.insightfullogic.java8.examples.chapter3.RefactorTest#allStringJoins

  • To run the test without NonDex:
    mvn test -Dtest=com.insightfullogic.java8.examples.chapter3.RefactorTest#allStringJoins

Expected results

  • This test should run successfully when run with NonDex.

Actual Result

  • We get the following failure:
-------------------------------------------------------------------------------
Test set: com.insightfullogic.java8.examples.chapter3.RefactorTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.029 sec <<< FAILURE! - in com.insightfullogic.java8.examples.chapter3.RefactorTest
allStringJoins(com.insightfullogic.java8.examples.chapter3.RefactorTest)  Time elapsed: 0.028 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[[Acknowledgement, Resolution]]> but was:<[[Resolution, Acknowledgement]]>
	at org.junit.Assert.assertEquals(Assert.java:115)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at com.insightfullogic.java8.examples.chapter3.RefactorTest.lambda$allStringJoins$0(RefactorTest.java:38)

Description of Fix

  • I added some code to convert the HashSet into a list and then used Collections.sort() to sort it before comparing to the hard-coded string value.

This test was passing when run on its own but flaky when run with other
tests.

Discovered with the nondex tool: https://github.com/TestingResearchIllinois/NonDex
Ran test on its own with command: mvn test -Dtest=com.insightfullogic.java8.examples.chapter3.RefactorTest#allStringJoins
Ran tests in groups with command: mvn edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=com.insightfullogic.java8.examples.chapter3.RefactorTest#allStringJoins

Fixed test by adding some code to sort the HashSet before comparing it
to a hard-coded string.
@jcoben jcoben changed the title [SWE637 FA23] Fix flaky test RefactorTest Fix flaky test RefactorTest Oct 19, 2023
@aswin-devil
Copy link

feab683

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants